Method for tracking sports participants, device for tracking sports participants, and system for tracking sports participants

ABSTRACT

A multi-mode tracking method according to the present disclosure includes receiving sensor signals from a plurality of positioning sensors located on a plurality of sport participants wherein the sensor signals each include a participant identifier and location data, receiving a sport image captured from a camera located near a playfield wherein the sport image includes at least a target participant among a plurality of sport participants on the playfield, detecting an occlusion related to the target participant in the sports image, determining the severity of the occlusion on the basis of a sensor signal received from a specific positioning sensor installed on a specific sport player located in a region of interest related to the occlusion, and determining a location of the sport participant on the basis of the severity of the occlusion.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 10-2020-0153137 filed on Nov. 16, 2020, and KoreanPatent Application No. 10-2020-0153138 filed on Nov. 16, 2020, thedisclosure of which is incorporated herein by reference in its entirety.

BACKGROUND 1. Field of the Invention

The present disclosure is related to a player tracking method, a playertracking device, and a player tracking system capable of trackingplayers' locations.

2. Discussion of Related Art

With the improvement of image analysis technology and positioningtechnology, it has become possible to analyze the movements or motionsof players during sport events, and thus sport analysis technology thatquantifies the performance of sport players has recently attractedattention. When quantifying the performance of a sport player, theanalysis of the player's location is considered as the most basicfactor.

As a representative method of calculating a sport player's location, amethod of calculating a position where a player is located in a stadiumfrom an image captured by a camera and a method of using positioninginstruments such as a Global Positioning System (GPS) device or a localpositioning system (LPS) device are known.

However, a positioning method using a video has several limitations dueto camera technology, including the inability to cope with a blockagesituation in which a player to be tracked is hidden by other players ina video, and a positioning method using positioning instruments such asa GPS module has fundamental problems due to the decrease in positioningaccuracy or the delay or reflection of signals which are caused by thelocations of satellites, so each method has its limitations insituations where high precision is required.

SUMMARY OF THE INVENTION

The present disclosure is directed to providing a player trackingmethod, a player tracking device, and a player tracking system withimproved accuracy of player location tracking.

A player tracking method according to the present disclosure may includereceiving sensor signals from a plurality of positioning sensors locatedon a plurality of sport participants wherein the sensor signals eachinclude a participant identifier and location data, receiving a sportimage captured by a camera located near a playfield wherein the sportimage includes at least a target participant among the plurality ofsport participants on the playfield, detecting an occlusion related tothe target participant in the sport image, determining the severity ofthe occlusion on the basis of sensor signals received from specificpositioning sensors installed on specific sport players located in aregion of interest related to the occlusion, determining a location ofthe sport participant on the basis of a pixel location related to thetarget participant in the sport image when it is not detected that theseverity of the occlusion is greater than a threshold value, anddetermining a location of the sport participant on the basis of locationdata of a sensor signal having a participant identifier indicating thetarget participant when it is detected that the severity of theocclusion is greater than the threshold value.

A player tracking method according to the present disclosure may includeacquiring image-based location data indicating a first location of asport participant defined in a first coordinate system from a sportimage which is captured by a camera located near a playfield and whichincludes sport participants on the playfield, wherein the image-basedlocation data is acquired based on a pixel location related to the sportparticipant in the sport image, detecting an occlusion event related tothe sport participant in the sport image, verifying the image-basedlocation data on the basis of a result of the detection, acquiring apositioning sensor-based location indicating a second location of thesport participant defined in a second coordinate system on the basis ofa sensor signal acquired from a positioning sensor located on the sportparticipant, acquiring a positioning sensor-based velocity indicating avelocity of the sport participant defined in the second coordinatesystem on the basis of the second location or the sensor signal,preparing a neural network for converting a location in the secondcoordinate system to the first coordinate system, wherein the neuralnetwork includes an input layer for receiving a location and a velocitycorresponding to the second coordinate system, an output layer foroutputting a result indicating a location value, and a hidden layerhaving a plurality of nodes connecting the input layer and the outputlayer, preparing a training set, wherein the training set includesimage-based location data determined to be valid and positioningsensor-based location data and positioning sensor-based velocity datacorresponding to the image-based location data determined to be valid;and training the neural network using the training set by inputting thepositioning sensor-based location data and the positioning sensor-basedvelocity data of the training set to the input layer and adjustingweight values of the nodes on the basis of a difference between thelocation value of the result and the image-based location datacorresponding to the input positioning sensor-based location data andpositioning sensor-based velocity data.

A player tracking method according to the present disclosure may includereceiving a sport image captured by a camera located near a playfield,wherein the sport image includes a sport participant on the playfield,receiving a sensor signal from a positioning sensor installed on thesport participant, detecting an occlusion event related to the sportparticipant in the sport image, determining a location of the sportparticipant on the basis of image-based location data indicating alocation defined in a first coordinate system when the occlusion eventis not detected, acquiring a positioning sensor-based locationindicating a location defined in a second coordinate system on the basisof the sensor signal when the occlusion event is detected, acquiring apositioning sensor-based velocity indicating a velocity defined in thesecond coordinate system on the basis of the sensor signal or thepositioning sensor-based location, and determining the location of thesport participant defined in the first coordinate system from thepositioning sensor-based location and the positioning sensor-basedvelocity using a neural network for converting a location in the secondcoordinate system to the first coordinate system, wherein the neuralnetwork may include an input layer for receiving a location and avelocity according to the second coordinate system, an output layer foroutputting a result indicating a location value corresponding to thefirst coordinate system, and a hidden laver having a plurality of nodesconnecting the input layer and the output layer and may be trained byadjusting weight values of the plurality of nodes using a location and avelocity which correspond to the second coordinate system and which arelabeled with the location corresponding to the first coordinate system.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location on the basisof the sensor signal, verifying the positioning sensor-based location onthe basis of a change in the positioning sensor-based location comparedto the previous positioning sensor-based location, determining alocation of the sport participant on the basis of the positioningsensor-based location when the positioning sensor-based location isvalid, acquiring a sport image captured by a camera located near aplayfield when the positioning sensor-based location is invalid, whereinthe sport image includes a sport participant on the playfield,predicting the validity of an image-based location acquired byprojecting a pixel of the sport participant in the sport image onto areference plane having the same height as the playfield, wherein thepredicting includes at least one of detecting an occlusion related tothe sport participant, detecting a vertical movement related to thesport participant, and computing a change in the image-based locationcompared to the previous image-based location, acquiring the image-basedlocation on the basis of a pixel location of the sport participant inthe sport image and determining the location of the sport participant onthe basis of the image-based location when the image-based location ispredicted to be valid, and determining the location of the sportparticipant on the basis of the positioning sensor-based location whenthe image-based location is predicted to be invalid.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location of the sportparticipant on the basis of the sensor signal, acquiring a sport imagecaptured by a camera located near a playfield, wherein the sport imageincludes a sport participant on the playfield, acquiring an image-basedlocation of the sport participant on the basis of a pixel location ofthe sport participant in the sport image, computing a disparity indexbetween the positioning sensor-based location and the image-basedlocation, wherein the disparity index is acquired from a differencebetween the positioning sensor-based location and the image-basedlocation, determining a location of the sport participant on the basisof a first location, which is one of the positioning sensor-basedlocation and the image-based location, when the disparity index issmaller than a predetermined first threshold value, acquiring a firstreliability index related to one of the positioning sensor-basedlocation and the image-based location when the disparity index isgreater than a predetermined second threshold value, acquiring a secondreliability index related to the other one of the positioningsensor-based location and the image-based location, determining thelocation of the sport participant on the basis of the first locationwhen the first reliability index is greater than the second reliabilityindex, and determining the location of the sport participant on thebasis of a second location, which is the other one of the positioningsensor-based location and the image-based location when the firstreliability index is smaller than the second reliability index.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location of the sportparticipant on the basis of the sensor signal, acquiring firstreliability information related to reliability of the positioningsensor-based location, acquiring a sport image captured by a cameralocated near a playfield, wherein the sport image includes a sportparticipant on the playfield, acquiring an image-based location of thesport participant on the basis of a pixel location of the sportparticipant in the sport image, acquiring second reliability informationrelated to reliability of the image-based location, wherein thereliability of the image-based location is related to at least one of anocclusion related to the sport participant, a vertical movement relatedto the sport participant, and a change in the image-based locationcompared to the previous image-based location, computing a weight valueon the basis of the first reliability information and the secondreliability information, wherein the weight value includes asensor-based weight value and an image-based weight value, and computinga location of the sport participant, wherein the location of the sportparticipant is acquired from the positioning sensor-based locationconsidering the sensor-based weight value and the image-based locationconsidering the image-based weight value.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location of the sportparticipant on the basis of the sensor signal, preparing a reliabilitymap corresponding to a playfield, wherein the reliability map includes aplurality of regions having reliability information including firstreliability information related to reliability of the positioningsensor-based location, acquiring a sport image captured by a cameralocated near the playfield, wherein the sport image includes a sportparticipant on the playfield, acquiring an image-based location of thesport participant on the basis of a pixel location of the sportparticipant in the sport image, determining a specific region occupiedby the sport participant on the basis of the positioning sensor-basedlocation, determining a first weight value and a second weight valueaccording to the first reliability information of the specific region;and determining a location of the sport participant on the basis of aweighted average of the positioning sensor-based location and theimage-based location computed in consideration of the first weight valueand the second weight value.

A player tracking method according to the present disclosure may includereceiving a plurality of sport images from a plurality of camerasinstalled at different locations near a playfield, wherein each of theplurality of sport images includes the playfield and at least some sportparticipants, detecting a target participant in the plurality of sportimages, selecting at least one candidate image from among the pluralityof sport images according to a result of the detection of the targetparticipant, detecting an occlusion related to the target participant inthe at least one candidate image, selecting at least one valid imagefrom the at least one candidate image according to a result of thedetection of the occlusion, and determining a location of the targetparticipant on the basis of a pixel location corresponding to the targetparticipant in the at least one valid image.

A player tracking method according to the present disclosure may includereceiving a plurality of sport images from a plurality of camerasinstalled at different locations near a playfield, wherein each of theplurality of sport images includes the playfield and at least some sportparticipants, detecting a target participant in the plurality of sportimages, selecting at least one candidate image from among the pluralityof sport images according to a result of the detection of the targetparticipant, detecting an occlusion related to the target participant inthe at least one candidate image, selecting at least one target imagefrom the at least one candidate image according to a result of thedetection of the occlusion, determining whether the sport participanthides another sport participant in the at least one candidate image orwhether the sport participant is hidden by another sport participant inthe at least one candidate image and selecting the target image from theat least one candidate image according to the determination when theocclusion is detected in all of the one or more candidate images, anddetermining a location of the target participant on the basis of a pixellocation corresponding to the target participant in the at least onetarget image.

A player tracking method according to the present disclosure may includeacquiring camera arrangement information including locations andorientations of a plurality of cameras located near a playfield,receiving sensor signals from a plurality of positioning sensors locatedon a plurality of sport participants, acquiring locations of theplurality of sport participants on the basis of the sensor signals,generating a virtual playfield including a plurality of fixed pointscorresponding to the plurality of cameras and a plurality of movingpoints corresponding to the plurality of sport participants, whereinlocations of the plurality of fixed points in the virtual playfield aredetermined based on the camera arrangement information and locations ofthe plurality of moving points in the virtual playfield are determinedbased on the locations of the plurality of sport participants, computinga relationship between the plurality of sport participants and theplurality of cameras on the basis of a relative location between theplurality of fixed points and the plurality of moving points, predictingan occlusion of the plurality of sport participants by the plurality ofcameras on the basis of relative locations between the plurality ofmoving points and angles between a plurality of virtual lines extendingfrom the plurality of fixed points to the plurality of moving points,and generating a matching table between the sport participant and theplurality of cameras on the basis of the occlusion prediction and therelationship.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentdisclosure will become more apparent to those of ordinary skill in theart by describing exemplary embodiments thereof in detail with referenceto the accompanying drawings, in which:

FIG. 1 is a diagram illustrating an exemplary conventional positioningmethod;

FIG. 2 is a diagram illustrating another exemplary conventionalpositioning method;

FIG. 3 is a schematic diagram illustrating a player tracking systemaccording to an embodiment of the present disclosure;

FIG. 4 is a block diagram of a player tracking server 1000 according tothe present disclosure;

FIGS. 5 to 8 are exemplary diagrams illustrating a method of the playertracking server determining or detecting whether an occlusion eventoccurs between a plurality of players in image data according to anembodiment of the present disclosure;

FIG. 9 is an exemplary diagram of a method of determining the validityof a player's location computed from image data according to the presentdisclosure;

FIG. 10 is an exemplary diagram illustrating a method of determining thevalidity of a player's location computed from image data according tothe present disclosure;

FIG. 11 is an exemplary diagram of a method of determining the validityof a player's location computed from positioning sensor data accordingto the present disclosure;

FIG. 12 is an exemplary diagram illustrating a method of determining thevalidity of a player's location computed from positioning sensor dataaccording to the present disclosure;

FIG. 13 is a diagram illustrating that an artificial neural network istrained to convert a location in a first coordinate system related topositioning sensor data into a location in a second coordinate systemrelated to image data;

FIG. 14 is a diagram illustrating that a location in the firstcoordinate system related to positioning sensor data is converted to alocation in the second coordinate system related to image data using thetrained artificial neural network;

FIG. 15 is a diagram illustrating an exemplary method of a playertracking server determining a player's location according to anembodiment of the present disclosure;

FIG. 16 is a flowchart illustrating a player tracking method accordingto an embodiment of the present disclosure;

FIG. 17 is a flowchart illustrating the player tracking method accordingto an embodiment of the present disclosure and is a diagram illustratingoperation S1300 of FIG. 16 in detail;

FIG. 18 is a flowchart illustrating a training process for an artificialneural network related to a coordinate conversion algorithm according toan embodiment of the present disclosure;

FIG. 19 is a flowchart illustrating a player tracking method throughcoordinate conversion using the trained artificial neural network;

FIG. 20 is a flowchart illustrating a player tracking method accordingto an embodiment of the present disclosure;

FIG. 21 is a flowchart illustrating the player tracking method accordingto an embodiment of the present disclosure and is a diagram illustratingoperation S4300 of FIG. 20 in detail;

FIG. 22 is a flowchart illustrating a player tracking method accordingto an embodiment of the present disclosure;

FIG. 23 is an example diagram of a first reliability map according to anembodiment of the present disclosure;

FIG. 24 is an example diagram of a second reliability map according toan embodiment of the present disclosure;

FIG. 25 is a flowchart illustrating a player tracking method accordingto an embodiment of the present disclosure;

FIG. 26 is a schematic diagram showing a player tracking systemutilizing a plurality of image capture devices according to anembodiment of the present disclosure;

FIG. 27 is a diagram illustrating an exemplary process of selecting avalid image according to an embodiment of the present disclosure;

FIG. 28 is a flowchart illustrating a player tracking system using aplurality of image capture devices according to an embodiment of thepresent disclosure;

FIG. 29 is an exemplary diagram showing a virtual playfield according toan embodiment of the present disclosure;

FIG. 30 is a flowchart illustrating a player tracking method using aplurality of image capture devices according to an embodiment of thepresent disclosure; and

FIG. 31 is a schematic diagram illustrating a method of computing asport participant's location through a player tracking method using aplurality of image capture devices according to an embodiment of thepresent disclosure.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The above objects, features, and advantages of the present disclosurewill become more apparent from the following detailed description takenin conjunction with the accompanying drawings. However, since thepresent disclosure may be variously modified and have several exemplaryembodiments, specific exemplary embodiments will be shown in theaccompanying drawings and described in detail.

Like reference numerals refer to like elements throughout thespecification. Further, like reference numerals will be used todesignate like elements within the same scope shown in the drawings ofthe embodiments, and a relevant description thereof will be omitted.

Detailed descriptions about well-known functions or configurationsassociated with the present disclosure will be omitted in order not tounnecessarily obscure the subject matter of the present disclosure. Itshould also be noted that, even though ordinal numbers (such as firstand second) are used in the following description, they are used only todistinguish similar elements.

The suffixes “module” and “unit” for elements used in the followingembodiments are given or used interchangeably only for facilitation ofpreparing this specification, and thus they are not assigned a specificmeaning or function.

As used herein, the singular forms “a,” “an,” and “one” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise.

It should be further understood that the terms “comprises” and/or“comprising” used herein specify the presence of stated features orcomponents, but do not preclude the presence or addition of one or moreother features or components.

Additionally, the sizes of components depicted in the drawings may beexaggerated or reduced for convenience of description. For example,since the sizes and thicknesses of components in the drawings arearbitrarily illustrated for convenience of description, the followingembodiments are not limited thereto.

When a certain embodiment may be implemented differently, a specificprocess order may be performed differently from the described order. Forexample, two consecutively described processes may be performedsubstantially at the same time or performed in the order opposite to thedescribed order.

In the following embodiments, when elements are connected to each other,the elements are connected to each other not only directly but alsoindirectly with other elements interposed therebetween.

For example, in the following embodiments, when elements areelectrically connected to each other, the elements are electricallyconnected to each other not only directly but also indirectly with otherelements interposed therebetween.

A player tracking method according to the present disclosure may includereceiving sensor signals from a plurality of positioning sensors locatedon a plurality of sport participants wherein the sensor signals eachinclude a participant identifier and location data, receiving a sportimage captured by a camera located near a playfield wherein the sportimage includes at least a target participant among the plurality ofsport participants on the playfield, detecting an occlusion related tothe target participant in the sport image, determining the severity ofthe occlusion on the basis of sensor signals received from specificpositioning sensors installed on specific sport players located in aregion of interest related to the occlusion, determining a location ofthe sport participant on the basis of a pixel location related to thetarget participant in the sport image when it is not determined that theseverity of the occlusion is greater than a threshold value, anddetermining a location of the sport participant on the basis of locationdata of a sensor signal having a participant identifier indicating thetarget participant when it is determined that the severity of theocclusion is greater than the threshold value.

The region of interest may be a predetermined region for the targetparticipant associated with the occlusion, and the determining of theseverity may include determining the severity of the occlusion on thebasis of the number of sensor signals located in the region of interest.

The determining of the severity may include assigning a value indicatinga severe occlusion for the severity when the number of sensor signals isgreater than the threshold value and assigning a value indicating a mildocclusion for the severity when the number of sensor signals is smallerthan the threshold value.

The determining of the severity may further include acquiring teaminformation indicating whether the target participant is on the sameteam as at least one sport participant different from the targetparticipant on the basis of a participant identifier corresponding tothe target participant and the sport participant, and the sportparticipant may hide or be hidden by the target participant.

The determining of the severity may include assigning the valueindicating the severe occlusion for the severity when the targetparticipant and the sport participant are on the same team and assigningthe value indicating the mild occlusion for the seventy when the targetparticipant and the sport participant are on different teams.

The team information may be acquired from pixel data corresponding tothe target participant and the sport participant in the sport image.

A player tracking method according to the present disclosure may includeacquiring image-based location data indicating a first location of asport participant defined in a first coordinate system from a sportimage which is captured by a camera located near a playfield and whichincludes sport participants on the playfield, wherein the image-basedlocation data is acquired based on a pixel location related to the sportparticipant in the sport image, detecting an occlusion event related tothe sport participant in the sport image, verifying the image-basedlocation data on the basis of a result of the detection, acquiring apositioning sensor-based location indicating a second location of thesport participant defined in a second coordinate system on the basis ofa sensor signal acquired from a positioning sensor located on the sportparticipant, acquiring a positioning sensor-based velocity indicating avelocity of the sport participant defined in the second coordinatesystem on the basis of the second location or the sensor signal,preparing a neural network for converting a location in the secondcoordinate system to the first coordinate system, wherein the neuralnetwork includes an input layer for receiving a location and a velocitycorresponding to the second coordinate system, an output layer foroutputting a result indicating a location value, and a hidden layerhaving a plurality of nodes connecting the input layer and the outputlayer, preparing a training set, wherein the training set includesimage-based location data determined to be valid and positioningsensor-based velocity data and positioning sensor-based location datacorresponding to the image-based location data determined to be valid;and training the neural network using the training set by inputting thepositioning sensor-based location data and positioning sensor-basedvelocity data of the training set to the input layer and adjustingweight values of the nodes on the basis of a difference between thelocation value of the result and the image-based location datacorresponding to the input positioning sensor-based location data andpositioning sensor-based velocity data.

The verifying may include determining that the image-based location datais valid when the occlusion event is not detected and determining thatthe image-based location data is invalid when the occlusion event isdetected.

The occlusion event may indicate that the number of sensor signals of atleast one sport participant in a region of interest related to theocclusion is greater than a threshold value, and the region of interestmay be a predetermined region for a region where the occlusion hasoccurred.

A player tracking method according to the present disclosure may includereceiving a sport image captured by a camera located near a playfield,wherein the sport image includes a sport participant on the playfield,receiving a sensor signal from a positioning sensor installed on thesport participant, detecting an occlusion event related to the sportparticipant in the sport image, determining a location of the sportparticipant on the basis of image-based location data indicating alocation defined in a first coordinate system when the occlusion eventis not detected, acquiring a positioning sensor-based locationindicating a location defined in a second coordinate system on the basisof the sensor signal when the occlusion event is detected, acquiring apositioning sensor-based velocity indicating a velocity defined in thesecond coordinate system on the basis of the sensor signal or thepositioning sensor-based location, and determining the location of thesport participant defined in the first coordinate system from thepositioning sensor-based location and the positioning sensor-basedvelocity using a neural network for converting the location in thesecond coordinate system to the first coordinate system, wherein theneural network may include an input layer for receiving a location and avelocity according to the second coordinate system, an output layer foroutputting a result indicating a location value corresponding to thefirst coordinate system, and a hidden layer having a plurality of nodesconnecting the input layer and the output layer and may be trained byadjusting weight values of the plurality of nodes using a location and avelocity which correspond to the second coordinate system and which arelabeled with the location corresponding to the first coordinate system.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location on the basisof the sensor signal, verifying the positioning sensor-based location onthe basis of a change in the positioning sensor-based location comparedto the previous positioning sensor-based location, determining alocation of the sport participant on the basis of the positioningsensor-based location when the positioning sensor-based location isvalid, acquiring a sport image captured by a camera located near aplayfield when the positioning sensor-based location is invalid, whereinthe sport image includes a sport participant on the playfield,predicting the validity of an image-based location acquired byprojecting a pixel of the sport participant in the sport image onto areference plane having the same height as the playfield, wherein thepredicting includes at least one of detecting an occlusion related tothe sport participant, detecting a vertical movement related to thesport participant, and computing a change in the image-based locationcompared to the previous image-based location, acquiring the image-basedlocation on the basis of a pixel location of the sport participant inthe sport image and determining the location of the sport participant onthe basis of the image-based location when the image-based location ispredicted to be valid, and determining the location of the sportparticipant on the basis of the positioning sensor-based location whenthe image-based location is predicted to be invalid.

The verifying of the sensor-based location may include determining thatthe sensor-based location is invalid when the change in the positioningsensor-based location compared to the previous positioning sensor-basedlocation is greater than a predetermined threshold value and determiningthat the sensor-based location is valid when the change in thepositioning sensor-based location compared to the previous positioningsensor-based location is smaller than a predetermined threshold value.

The verifying of the positioning sensor-based location may includefurther considering sensor signal reliability-related informationincluded in the sensor signal.

The sensor signal reliability-related information included in the sensorsignal may include at least one of a degree of polarization (DoP) and asignal-to-noise ratio (SNR).

The vertical movement related to the sport participant may be detectedbased on a movement of the sport participant in a directionperpendicular to the reference plane acquired from an inertial sensorworn by the sport participant.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location of the sportparticipant on the basis of the sensor signal, acquiring a sport imagecaptured by a camera located near a playfield, wherein the sport imageincludes a sport participant on the playfield, acquiring an image-basedlocation of the sport participant on the basis of a pixel location ofthe sport participant in the sport image, computing a disparity indexbetween the positioning sensor-based location and the image-basedlocation, wherein the disparity index is acquired from a differencebetween the positioning sensor-based location and the image-basedlocation, determining a location of the sport participant on the basisof a first location, which is one of the positioning sensor-basedlocation and the image-based location, when the disparity index issmaller than a predetermined first threshold value, acquiring a firstreliability index related to one of the positioning sensor-basedlocation and the image-based location when the disparity index isgreater than a predetermined second threshold value, acquiring a secondreliability index related to the other one of the positioningsensor-based location and the image-based location, determining thelocation of the sport participant on the basis of the first locationwhen the first reliability index is greater than the second reliabilityindex, and determining the location of the sport participant on thebasis of a second location, which is the other one of the positioningsensor-based location and the image-based location when the firstreliability index is smaller than the second reliability index.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location of the sportparticipant on the basis of the sensor signal, acquiring firstreliability information related to the reliability of the positioningsensor-based location, acquiring a sport image captured by a cameralocated near a playfield, wherein the sport image includes a sportparticipant on the playfield, acquiring an image-based location of thesport participant on the basis of a pixel location of the sportparticipant in the sport image, acquiring second reliability informationrelated to the reliability of the image-based location, wherein thereliability of the image-based location is related to at least one of anocclusion related to the sport participant, a vertical movement relatedto the sport participant, and a change in the image-based locationcompared to the previous image-based location, computing a weight valueon the basis of the first reliability information and the secondreliability information, wherein the weight value includes asensor-based weight value and an image-based weight value, and computinga location of the sport participant, wherein the location of the sportparticipant is acquired from the positioning sensor-based locationconsidering the sensor-based weight value and the image-based locationconsidering the image-based weight value.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location of the sportparticipant on the basis of the sensor signal, acquiring firstreliability information related to the reliability of the positioningsensor-based location, acquiring a sport image captured by a cameralocated near a playfield, wherein the sport image includes a sportparticipant on the playfield, acquiring an image-based location of thesport participant on the basis of a pixel location of the sportparticipant in the sport image, acquiring second reliability informationrelated to the reliability of the image-based location, wherein thereliability of the image-based location is related to at least one of anocclusion related to the sport participant, a vertical movement relatedto the sport participant, and a change in the image-based locationcompared to the previous image-based location, computing an weight valueon the basis of the first reliability information and the secondreliability information, wherein the weight value includes asensor-based weight value and an image-based weight value, and computinga location of the sport participant, wherein the location of the sportparticipant is acquired from the positioning sensor-based locationconsidering the sensor-based weight value and the image-based locationconsidering the image-based weight value.

The reliability of the positioning sensor-based location may be relatedto at least one of the computed change in the positioning sensor-basedlocation compared to the previous positioning sensor-based location andthe sensor signal reliability-related information included in the sensorsignal.

The vertical movement related to the sport participant may be detectedbased on a movement of the sport participant which is in a directionperpendicular to the reference plane with the same height as theplayfield and which is acquired from an inertial sensor worn by thesport participant.

The reliability of the positioning sensor-based location and thereliability of the image-based location may be related to a disparityindex between the positioning sensor-based location and the image-basedlocation, and the disparity index may be acquired from a differencebetween the positioning sensor-based location and the image-basedlocation.

A player tracking method according to the present disclosure may includereceiving a sensor signal from a positioning sensor located on a sportparticipant, acquiring a positioning sensor-based location of the sportparticipant on the basis of the sensor signal, preparing a reliabilitymap corresponding to a playfield, wherein the reliability map includes aplurality of regions having reliability information including firstreliability information related to the reliability of the positioningsensor-based location, acquiring a sport image captured by a cameralocated near the playfield, wherein the sport image includes a sportparticipant on the playfield, acquiring an image-based location of thesport participant on the basis of a pixel location of the sportparticipant in the sport image, determining a specific region occupiedby the sport participant on the basis of the positioning sensor-basedlocation; determining a first weight value and a second weight valueaccording to the first reliability information of the specific region;and determining a location of the sport participant on the basis of aweighted average of the positioning sensor-based location and theimage-based location computed in consideration of the first weight valueand the second weight value.

The first reliability information related to the reliability of thepositioning sensor-based location may be related to at least one of thecomputed change in the positioning sensor-based location compared to theprevious positioning sensor-based location and the sensor signalreliability-related information included in the sensor signal.

The reliability map may include a plurality of regions havingreliability information including second reliability information relatedto the reliability of the image-based location.

The second reliability information related to the reliability of theimage-based location may be related to at least one of a detectedocclusion related to the sport participant, a detected vertical movementrelated to the sport participant, and a computed change in theimage-based location compared to the previous image-based location.

The first weight value and the second weight value may be determinedaccording to the second reliability information of the specific region.

A player tracking method according to the present disclosure may includereceiving a plurality of sport images from a plurality of camerasinstalled at different locations near a playfield, wherein each of theplurality of sport images includes the playfield and at least some sportparticipants, detecting a target participant in the plurality of sportimages, selecting at least one candidate image from among the pluralityof sport images according to a result of the detection of the targetparticipant, detecting an occlusion related to the target participant inthe at least one candidate image, selecting at least one valid imagefrom the at least one candidate image according to a result of thedetection of the occlusion, and determining a location of the targetparticipant on the basis of a pixel location corresponding to the targetparticipant in the at least one valid image.

When the at least one valid image includes a first valid image and asecond valid image, the determining of the location of the targetparticipant may include acquiring a first location of the targetparticipant on the basis of a pixel location corresponding to the targetparticipant in the first valid image, acquiring a second location of thetarget participant on the basis of a pixel location corresponding to thetarget participant in the second valid image, and computing a locationof the target participant on the basis of both of the first location andthe second location.

The determining of the location of the target participant may includeselecting a single image from the at least one valid image on the basisof a first condition related to a relative location between the targetparticipant and a camera that captures the at least one valid image anda second condition related to a lens distortion parameter of the camerathat captures the at least one valid image and computing a location ofthe target participant on the basis of the single image.

A player tracking method according to the present disclosure may includereceiving a plurality of sport images from a plurality of camerasinstalled at different locations near a playfield, wherein each of theplurality of sport images includes the playfield and at least some sportparticipants, detecting a target participant in the plurality of sportimages, selecting at least one candidate image from among the pluralityof sport images according to a result of the detection of the targetparticipant, detecting an occlusion related to the target participant inthe at least one candidate image, selecting at least one target imagefrom the at least one candidate image according to a result of thedetection of the occlusion, determining whether the sport participanthides or is hidden by another sport participant in the at least onecandidate image and selecting the target image from the at least onecandidate image according to the determination when the occlusion isdetected in all of the one or more candidate images, and determining alocation of the target participant on the basis of a pixel locationcorresponding to the target participant in the at least one targetimage.

The at least one target image may be selected from among the at leastone candidate image in which the sport participant hides the other sportparticipant in the at least one candidate image.

When the at least one target image includes a first target image and asecond target image, the determining of the location of the targetparticipant may include acquiring a first location of the targetparticipant on the basis of a pixel location corresponding to the targetparticipant in the first target image, acquiring a second location ofthe target participant on the basis of a pixel location corresponding tothe target participant in the second target image, and computing alocation of the target participant on the basis of both of the firstlocation and the second location.

The determining of the location of the target participant may includeselecting a single image from the at least one target image on the basisof a first condition related to a relative location between the targetparticipant and a camera that captures the at least one target image anda second condition related to a lens distortion parameter of the camerathat captures the at least one target image and computing a location ofthe target participant on the basis of the single image.

A player tracking method according to the present disclosure may includeacquiring camera arrangement information including locations andorientations of a plurality of cameras located near a playfield,receiving sensor signals from a plurality of positioning sensors locatedon a plurality of sport participants, acquiring locations of theplurality of sport participants on the basis of the sensor signals,generating a virtual playfield including a plurality of fixed pointscorresponding to the plurality of cameras and a plurality of movingpoints corresponding to the plurality of sport participants, whereinlocations of the plurality of fixed points on the virtual playfield aredetermined based on the camera arrangement information and locations ofthe plurality of moving points on the virtual playfield are determinedbased on the locations of the plurality of sport participants, computinga relationship between the plurality of sport participants and theplurality of cameras on the basis of a relative location between theplurality of fixed points and the plurality of moving points, predictingan occlusion of the plurality of sport participants by the plurality ofcameras on the basis of relative locations between the plurality ofmoving points and angles between a plurality of virtual lines extendingfrom the plurality of fixed points to the plurality of moving points,and generating a matching table between the sport participant and theplurality of cameras on the basis of the occlusion prediction and therelationship.

The following description will refer to FIG. 1 . FIG. 1 is a diagramshowing an exemplary conventional positioning method. Specifically, FIG.1 shows a specific situation of a positioning method that computes apoint where a player is located for an image captured through a camera.

According to a video-based positioning method, only when a player to betracked is recognized in a video can the player's location be accuratelycomputed.

For example, a player located in a first region 10 of FIG. 1 may berecognized, and thus the player's location can be computed for tracking.

However, a player located in a second region 12 of FIG. 1 may bedifficult to recognize because an occlusion event occurs. Specifically,since a plurality of players are concentrated in the second region 12,an occlusion in which a player to be tracked is covered by anotherplayer in a video may occur. Therefore, the location of the player to betracked may not be accurately acquired because the player is hidden byanother player.

In other words, the video-based positioning method may not be able tocope with the occlusion situation in which a player to be tracked ishidden by another player in a video.

The following description will refer to FIG. 2 . FIG. 2 is a diagramillustrating another exemplary conventional positioning method.Specifically, FIG. 2 is a diagram showing a positioning method thatcomputes a point where a player is located using positioning instrumentssuch as a GPS module.

Specifically, a GPS module-based positioning method computes a locationof a player to be tracked depending on a signal transmitted from asatellite. However, signals transmitted from satellites may be greatlyaffected by structures around the player to be tracked.

For example, GPS signals transmitted from some satellites 24 and 26 ofFIG. 2 may be transmitted to the inside of a stadium without beingaffected by the structures around the player to be tracked. However, GPSsignals transmitted from some satellites 22 and 28 of FIG. 2 may not beable to reach the inside of the stadium due to the influence of thestructures around the player to be tracked. At this time, when the GPSsignals transmitted from some satellites 22 and 28 are affected by thesurrounding structures, there may be an error in the player's locationcomputed from the GPS signals.

Also, since the degree to which a GPS signal is affected by surroundingstructures is different for each region of a playfield in the stadium,the GPS module-based positioning method has a problem in thatreliability is different for each region of the playfield in thestadium.

Herein, a player tracking device, a player tracking system, and a playertracking method for overcoming the limitations of the positioning methodusing videos or the positioning method using positioning instrumentssuch as a GPS module or a local positioning system (LPS) module will bedescribed.

The player tracking device, the player tracking system, and the playertracking method of the present disclosure will be described below. Here,player tracking may be performed using positioning sensor data obtainedfrom a positioning sensor mounted on a player or image data acquiredthrough at least one camera. The present disclosure discloses atechnique of fusing the above-described data or a technique of usingmultiple pieces of image data in order to improve the accuracy of theplayer's location calculated from such basic data.

The player tracking method, the player tracking device, and the playertracking system according to an embodiment of the present disclosurewill be described below.

Player tracking according to this embodiment may be performed using bothimage data and positioning sensor data. Player tracking using image datamay compute a player's location on the basis of pixel data related tothe player's location in image data, and player tracking usingpositioning sensor data may compute a player's location on the basis ofdata such as latitude and longitude related to the player's locationincluded in the positioning sensor data. In this case, when the playertracking using the image data is incorrect, the player tracking may besupplemented by the positioning sensor data. When the player trackingusing the positioning sensor data is incorrect, the player tracking maybe supplemented by the image data.

FIG. 3 is a schematic diagram illustrating the player tracking device,the player tracking system, and the player tracking method according toan embodiment of the present disclosure.

Referring to FIG. 3 , the player tracking device, the player trackingsystem, and the player tracking method according to this embodiment mayinclude tracking a player's location using both image data andpositioning sensor data.

A player tracking system 100 according to an embodiment of the presentdisclosure will be described below with reference to FIG. 3 .

Referring to FIG. 3 , the player tracking system 100 may include apositioning sensor device 200, an image capture device 300, anintermediate server 400, and a player tracking device 1000.

The positioning sensor device 200 of the player tracking system 100 maybe a positioning instrument such as a GPS module and an LPS module.

As an example, the positioning sensor device 200 may be a GPS module210, and the GPS module may receive radio waves transmitted from asatellite in association with the Global Navigation System (GNSS) andtransmit data on the location of the positioning sensor device 200 tothe player tracking device 1000.

As another example, the positioning sensor device 200 may be an LPSmodule 220, and the LPS module may receive radio waves transmitted froma beacon 4 in association with an LPS and transmit data regarding thelocation of the positioning sensor device 200. The beacon 4 may beplaced in the vicinity of the playfield and in various locations.

The image capture device 300 of the player tracking system 100 may be acamera having any suitable form and function. The image capture device300 may transmit image data acquired by capturing a sport participant 1,a playfield line 2, and a playfield 3 to the player tracking device1000.

The player tracking device 1000 of the player tracking system 100 mayacquire image data and positioning sensor data. Also, the playertracking device 1000 may acquire an image-based location from the imagedata and may acquire a positioning sensor-based location from thepositioning sensor data. Also, the player tracking device 1000 maydetermine the validity of the image-based location and the positioningsensor-based location and may determine a location of a sportparticipant with high accuracy.

The player tracking device 1000 may be implemented as a server device ofany suitable type. Hereinafter, the player tracking device 1000 isreferred to as the player tracking server 1000. However, this is justfor convenience of description, and the player tracking device 1000 isnot necessarily implemented in the form of a server.

Meanwhile, the player tracking system 100 may further include anintermediate server 400 or a repeater. In other words, the playertracking server 1000 of the player tracking system 100 may be providedin a single form or may be provided in a separate form rather than inthe single form.

In this case, data transmitted from the positioning sensor device 200and the image capture device 300 may be transmitted directly to theplayer tracking server 1000 and may also be transmitted indirectly tothe player tracking server 1000 via the intermediate server 400. Whendata is transmitted via the intermediate server 400, the data may betransferred to a portable electronic device 420 through a dockingstation 410. Also, the data may be transferred from the portableelectronic device 420 to the player tracking server 1000 using anyappropriate communication method.

The player tracking server 1000 according to an embodiment of thepresent disclosure will be described below.

According to an embodiment of the present disclosure, the playertracking server 1000 may acquire image data from the image capturedevice 300 in order to determine a location of a sport participant.Also, the player tracking server 1000 may acquire positioning sensordata from a positioning sensor device in order to determine a locationof a sport participant. Also, the player tracking server 1000 maycompute an image-based location from the acquired image data. Also, theplayer tracking server 1000 may compute a positioning sensor-basedlocation from the acquired positioning sensor data. Also, the playertracking server 1000 may determine the location of the sport participanton the basis of the computed image-based location and positioningsensor-based location.

The following description will refer to FIG. 4 . FIG. 4 is a blockdiagram of the player tracking server 1000 according to an embodiment ofthe present disclosure.

Referring to FIG. 4 , the player tracking server 1000 may include acommunication module 1100, a memory 1200, and a controller 1300.

Components of the player tracking server 1000 according to an embodimentof the present disclosure will be described below.

The communication module 1100 may communicate with an externalapparatus. The player tracking server 1000 may transmit or receive datato or from an external server including the image capture device 300,the positioning sensor device 200, or the intermediate server 400through the communication module 1100.

For example, the player tracking server 1000 may receive image dataacquired from the image capture device 300 or positioning sensor dataacquired from the positioning sensor device 200 through thecommunication module 1100. As another example, the player trackingserver 1000 may access the Internet through the communication module1100 to upload the image data or the positioning sensor data.

The communication module 1100 is largely divided into a wired-typemodule and a wireless-type module. Since the wired-type module and thewireless-type module each have advantages and disadvantages, in somecases, the player tracking server 1000 may be provided with both of thewired-type module and the wireless-type module.

Here, the wired-type module may use, for example, local area network(LAN) or universal serial bus (USB) communication or other schemes.

Also, here, the wireless-type module may mainly use a wireless personalarea network (WPAN)-based communication scheme such as Bluetooth orZigbee. However, since a wireless communication protocol is not limitedthereto, the wireless-type communication module may use a wireless localarea network (WLAN)-based communication scheme such as Wi-Fi or otherknown communication schemes.

The memory 1200 may store various kinds of information. Various kinds ofdata may be temporarily or semi-permanently stored in the memory 1200.Examples of the memory 1200 may include a hard disk drive (HDD), asolid-state drive (SSD), a flash memory, a read-only memory (ROM), arandom access memory (RAM), etc.

The memory 1200 may be built into, or detachable from, the playertracking server 1000. Various kinds of data needed for the operation ofthe player tracking server 1000, in addition to an operating system (OS)for driving the player tracking server 1000 or a program for operatingthe components of the player tracking server 1000, may be stored in thememory 1200. For example, image data acquired in the form of anintensity map and an RGB map from the image capture device 300 andpositioning sensor data acquired from the positioning sensor device 200may be stored in the memory 1200.

The controller 1300 may control the overall operation of the playertracking server 1000. For example, the controller 1300 may load aprogram for the operating of the player tracking server 1000 from thememory 1200 and execute the program.

The controller 1300 may be implemented as a central processing unit(CPU) or the like with hardware, software, or a combination thereof. Thecontroller 1300 may be provided in the form of an electronic circuit forprocessing an electrical signal to perform a control function whenimplemented with hardware and may be provided in the form of a programor code for driving a hardware circuit when implemented with software.

Some operations performed by an example of the player tracking system100 in association with the player tracking method will be described indetail below.

The player tracking system 100 according to an embodiment of the presentdisclosure may acquire image data and positioning sensor data.

The player tracking server 1000 may acquire image data and positioningsensor data. Specifically, the player tracking server 1000 may acquireimage data from the image capture device 300 through the communicationmodule 1100. Also, the player tracking server 1000 may acquireinformation related to the image capture device 300 from the imagecapture device 300 through the communication module 1100. Also, theplayer tracking server 1000 may acquire positioning sensor data from thepositioning sensor device 200 through the communication module 1100.

The image data may include pixel information related to a plurality ofsport participants, playfield lines, playfield points, a ball, and thelike of a sport game and sport game progress information (elapsed time,image data capture time, and the like).

The information related to the image capture device 300 may includearrangement information related to the orientation and location of acamera. Alternatively, among the information related to the imagecapture device 300, the arrangement information related to the locationof the camera may be acquired by the player tracking server 1000 withany appropriate method.

The positioning sensor data may include location-related data of apositioning sensor device and reliability-related data of a sensorsignal.

The player tracking system 100 according to an embodiment of the presentdisclosure may resample the image data or remove noise.

Specifically, the player tracking server 1000 may be controlled toperform an operation of resampling the image data. Specifically, theplayer tracking server 1000 may perform an operation of resampling theimage data at a specific frame rate. Through the resampling operation ofthe player tracking server 1000, the player tracking server 1000 mayadjust time synchronization with positioning sensor data, which will bedescribed below. The specific frame rate may be any suitable frame rate(e.g., 10 frames).

Also, the player tracking server 1000 may be controlled to perform anoperation of removing noise from the image data. Specifically, imagedata captured by the image capture device may have distortion. Forexample, the image data may have distortion caused by lens distortion orhave distortion caused by the orientation (angle) of a camera or adistance from a camera. Therefore, the player tracking server 1000 mayperform an operation of correcting the above-described distortions ofthe image data.

The player tracking system 100 according to an embodiment of the presentdisclosure may perform an operation of correlating the image data withthe positioning sensor data. Specifically, the player tracking system100 may adjust time synchronization with the image data and thepositioning sensor data.

Specifically, the controller 1300 of the player tracking server 1000 maybe provided to correlate the image data with the positioning sensor datausing time variables. In other words, the controller 1300 of the playertracking server 1000 may be implemented to adjust time synchronizationbetween the image data and the positioning sensor data.

As an example, the time synchronization between the image data and thepositioning sensor data may be adjusted based on a specific time pointof the sport game.

In this case, the specific time point of the sport game may be acharacteristic time point of the sport game. For example, the specifictime point of the sport game may be a start time point of the game.

Also, when there is a break time in the sport game, the specific timepoint of the sport game may be a restart time point of the sport game.As an example, in the case of a soccer game, there is a half timebetween the first half and the second half, and in this case, the timesynchronization between the image data and the positioning sensor datamay be adjusted based on the start time of the first half and the starttime of the second half. At the start time point of the first half andthe start time point of the second half, some or more sport participantsstart to move, and the velocity of the sport participants may changeaccordingly. Therefore, the time synchronization between the image dataand the positioning sensor data may be adjusted in consideration of thechange in locations or velocitys of the sport participants at the starttime of the first half and the start time of the second half.

For example, the time synchronization between the image data and thepositioning sensor data may be adjusted based on a time point at whichthe velocity data of at least one sport participant acquired orcalculated from the image data matches the velocity data of at least onesport participant acquired or calculated from the positioning sensordata.

In this case, the number of sport participants subject to a comparisonbetween the velocity data from the image data and the velocity data fromthe positioning sensor data may be any appropriate number. In otherwords, the adjustment of the time synchronization through theabove-described comparison of the velocity data may be performed for allor some of the sport participants.

As an example, the time synchronization between the image data and thepositioning sensor data may be adjusted in consideration of a time pointat which the velocity change acquired from the positioning sensor datais detected. As a specific example, most of the sport participants startto move a predetermined time after the start point of the first half orthe second half, and thus the velocitys of the sport participants maychange. The positioning sensor device may acquire data related to thechange in locations of the sport participants and may detect a timepoint at which the velocity change acquired from the data related to thelocation change is started. According to an example, the timesynchronization between the image data and the positioning sensor datamay be adjusted based on a time point before a specific time point(e.g., more than 0 seconds to less than 2 seconds) based on the timepoint at which the velocity change is detected.

In this case, in association with the start time point of the velocitychange, when changes in velocity of a predetermined threshold number ormore of sport participants are detected, a time point at which thechanges are detected may be determined as a start time point of thevelocity changes.

Also, any suitable calculation method such as the Wasserstein distancecalculation method may be used to adjust the time synchronization.

Also, for accuracy, the time synchronization between the image data andthe positioning sensor data needs to be adjusted not only at the starttime point of the game but also during the game.

According to an embodiment of the present disclosure, the timesynchronization between the image data and the positioning sensor datamay be adjusted based on time points corresponding to a positionalrelationship between a plurality of sport participants acquired from theimage data at an arbitrary time point during the game and a positionalrelationship between a plurality of sport participants acquired from thepositioning sensor data.

As an example, the positioning sensor data may include an identifier ofeach of the sport participants, and location data of at least one sportparticipant identified using the identifier may be acquired from thepositioning sensor data.

Also, the at least one identified sport participant of the positioningsensor data may be distinguishable or identifiable in the image data,and the location data of the at least one identified sport participantmay be acquired from the image data.

In this case, a connective relationship between sport participants maybe acquired based on the location data of the at least one sportparticipant acquired from the positioning sensor data, and also aconnective relationship between sport participants may be acquired basedon the location data of the at least one sport participant acquired fromthe image data. In this case, the time synchronization between the imagedata and the positioning sensor data may be adjusted based on a timepoint at which the connective relationship between the sportparticipants acquired from the positioning sensor data and theconnective relationship between the sport participants acquired from theimage data match each other exactly or within an error range. Forexample, the connective relationship between the sport participants maybe expressed as a line or plane connecting the pieces of location dataof the sport participants acquired from the image data or thepositioning sensor data.

When the connective relationships between the sport participants areexpressed as lines, the time synchronization between the image data andthe positioning sensor data may be adjusted in consideration of whetherthe lengths or directions of the lines match each other.

When the connective relationships between the sport participants areexpressed as planes, the time synchronization between the image data andthe positioning sensor data may be adjusted in consideration of whetherthe shapes of figures consisting of the outlines of the planes matcheach other.

The player tracking system 100 according to an embodiment of the presentdisclosure may perform appropriate data processing such as noiseremoval, amplification, and filtering on positioning sensor dataacquired from the positioning sensor device 200. Specifically, theplayer tracking server 1000 may be provided to process theabove-described data processing.

The player tracking system 100 according to an embodiment of the presentdisclosure may change the format of image data. Specifically, the playertracking server 1000 may convert the format of image data received fromthe image capture device 300 from an RGB map to an intensity map. Inthis case, data related to the RGB map may be stored in the memory 1200of the player tracking server 1000. Data related to the RGB map of theimage data stored in the memory 1200 of the player tracking server 1000may be used by the player tracking server 1000, which will be describedbelow, to detect an occlusion event between players or determine theseverity of an occlusion event. By changing the format of image data,image data may be basically tracked in the form of an intensity map, butdata included in an RGB map may be used only when necessary. Thus, it ispossible to reduce the amount of computation of the player trackingserver 1000. Also, since the amount of computation is reduced, thecomputation velocity of player tracking may increase.

The player tracking system 100 according to an embodiment of the presentdisclosure may recognize and detect a playfield 3 included in the imagedata. Specifically, the player tracking server 1000 may perform anoperation of recognizing and detecting a playfield included in the imagedata in consideration of a reference point of the playfield.

For example, the player tracking server 1000 may recognize and detect aplayfield included in the image data in consideration of referencepoints such as a corner flag, a half line, a goal line, a touch line,and a penalty mark of the playfield. However, the above-describedreference points are just examples, and a playfield included in imagedata may be recognized and detected in consideration of any appropriateregion, line, point, or the like which is included in the playfield andwhich is capable of allowing the playfield to be recognized anddetected.

The player tracking system 100 according to an embodiment of the presentdisclosure may perform an image stitching operation.

Specifically, the player tracking server 1000 of the player trackingsystem 100 may be provided to perform image stitching from at least onepiece of image data to single image data.

As an example, image data may be acquired from at least one or moreimage capture devices disposed at different locations in the vicinity ofthe playfield. In this case, the player tracking server 1000 maygenerate a high-resolution image and a panoramic image by stitching atleast one or more pieces of image data acquired from at least one ormore image capture devices at the same time point into one piece ofimage data. In this case, when one or more pieces of image data arestitched into one piece of image data, the image data stitchingoperation may be performed based on a reference point included in anoverlapping portion between the pieces of image data. The “referencepoint” included in the overlapping portion between the pieces of imagedata may be a point such as a penalty mark or a corner flag included inthe image data. The “reference point” included in the overlappingportion between the pieces of image data may be a line such as a penaltyline, a goal line, a touch line, or a center line included in the imagedata. The “reference point” included in the overlapping portion betweenthe pieces of image data may be a region such as a center circleincluded in the image data. However, this is just an example, and anysuitable reference point may be used to perform a stitching operationbetween pieces of image data.

The player tracking system 100 according to an embodiment of the presentdisclosure may acquire an image-based location from the image data.

Specifically, the player tracking server 1000 may analyze the image datato compute a location of a sport participant included in the image data.

As an example, the player tracking server 1000 may convert image datafrom a perspective view form to a top view form in order to compute thelocation of the sport participant. In this case, the player trackingserver 1000 may consider an installation position of the image capturedevice that captures the image data or a relative location between sportparticipants acquired by the image capture device in order to convertcoordinates from the perspective view form to the top view form. Also,the player tracking server 1000 may convert image data in a perspectiveview form to image data in a top view form with respect to a plane thatis substantially the same as the playfield.

Specifically, the player tracking server 1000 may analyze the image datato compute a location of a sport participant included in the image datain the top view form. For example, the player tracking server 1000 maycompute the location of the sport participant on the basis of a locationof a pixel corresponding to the sport participant of the top-view imagedata. Preferably, the player tracking server 1000 may compute thelocation of the sport participant on the basis of a location of a pixelcorresponding to a foot of the sport participant.

As described above, although the description was focused on the playertracking server 1000 computing a player's location on the basis of imagedata in the top view form, this is just an example. For example, theplayer tracking server 1000 may compute a player's location by acquiringa location of a pixel corresponding to a sport participant (e.g., foot)in image data in a form other than the top view form (e.g., in theperspective view form) and then projecting the pixel onto a plane thatis substantially the same as the playfield in consideration of theheight and the installation angle of the image capture device.

As another example, in tracking the sport participant by analyzing theimage data to compute the location of the sport participant, variousartificial neural networks such as CSRT, fast RCNN, and Depp sort may beused. However, the above-described artificial neural network is just anexample, and any suitable algorithm or artificial neural network may beused to track sport participants.

For example, a sport participant included in image data may be segmentedthrough the above-described artificial neural network, and the sportparticipant may be detected. When the sport participant is detected, abounding box may be generated, and the player tracking server 1000 maycompute the location of the sport participant on the basis of thebounding box. Preferably, the player tracking server 1000 may computethe location of the sport participant on the basis of the bottom line ofthe bounding box.

As another example, a “foot” of the sport participant included in theimage data may be detected through the above-described artificial neuralnetwork, and the player tracking server 1000 may compute the location ofthe sport participant on the basis of a location of a pixelcorresponding to the “foot” of the sport participant.

The player tracking system 100 according to an embodiment of the presentdisclosure may acquire a positioning sensor-based location frompositioning sensor data. Specifically, the player tracking server 1000may compute a player's location in consideration of data related to theplayer's location included in the positioning sensor data. For example,the player tracking server 1000 may compute or acquire a positioningsensor-based location on the basis of positioning sensor data throughtriangulation.

Here, the positioning sensor data may include data related to theplayer's location. Alternatively, the positioning sensor-based locationmay be acquired from data related to velocity or acceleration includedin the positioning sensor data.

Also, the player tracking server 1000 may be provided to compute thevelocity or acceleration of the sport participant from location-relateddata of the image data and location-related data of the positioningsensor data using a mathematical method such as calculus.

The player tracking system 100 according to an embodiment of the presentdisclosure may determine the validity of the acquired image-basedlocation.

Specifically, the player tracking server 1000 may evaluate the validityof the image-based location of the player computed from the image data.More specifically, the player tracking server 1000 may be provided toconsider at least one of the presence of a detected occlusion event, theseverity of an occlusion event, a vertical movement of a player, and aninternal disparity of a player's location computed from image data inorder to evaluate the validity of the image-based location.

The following description will refer to FIGS. 5 to 8 . FIGS. 5 to 8 areexemplary views illustrating a method of the player tracking server 1000determining or detecting whether an occlusion event occurs between aplurality of players in image data according to an embodiment of thepresent disclosure.

As an example, the player tracking server 1000 may evaluate the validityof the player's location computed from the image data in considerationof whether an occlusion event is detected between a plurality of playersin the image data. Here, the occlusion event may refer to an event inwhich the plurality of players at least partially overlap.

For example, the player tracking server 1000 may detect an occlusionevent on the basis of image data in the form of an intensity map or anRGB map. For example, the player tracking server 1000 may recognize thesport participants included in the image data on the basis of anysuitable artificial intelligence network as described above, and as aresult of the recognition, a bounding box may be generated for eachsport participant. In this case, the player tracking server 1000 maydetermine whether an occlusion has occurred in consideration of thedegree to which the bounding boxes of the plurality of players overlap.

As an example, a threshold value related to the overlapping degree ofthe bounding boxes may be preset, and the player tracking server 1000may determine that the occlusion event has occurred when the overlappingdegree of the bounding boxes of the players exceeds the threshold value.On the contrary, the player tracking server 1000 may determine that noocclusion event has occurred when the bounding boxes of the players donot overlap or when the overlapping degree of the bounding boxes is lessthan the threshold value.

In detail, the following description will refer to FIG. 5 . FIG. 5 is adiagram showing that an occlusion is detected or determined according tothe size or proportion of an overlapping region between a first boundingbox BB1 corresponding to a first sport participant recognized from imagedata and a second bounding box BB2 corresponding to a second sportparticipant recognized from image data.

Referring to FIG. 5A, there may be a situation where the first boundingbox BB1 and the second bounding box BB2 overlap. In this case, theplayer tracking server 1000 may determine that no occlusion event hasoccurred.

Referring to FIGS. 5B, 5C, and 5D, there may be a situation where thefirst bounding box BB1 and the second bounding box BB2 partially do notoverlap. In this case, the player tracking server 1000 may determinewhether an occlusion event has occurred on the basis of a first presetthreshold value for the size or proportion of a region where the firstbounding box BB1 and the second bounding box BB2 overlap.

For example, FIG. 5B may be a diagram showing that the size orproportion of the region where the first bounding box BB1 and the secondbounding box BB2 overlap is smaller than the first preset thresholdvalue. In this case, the player tracking server 1000 may determine thatno occlusion event has occurred between the first sport participant andthe second sport participant.

As another example, FIG. 5C may be a diagram showing that the size orproportion of the region where the first bounding box BB1 and the secondbounding box BB2 overlap exceeds the first preset threshold value. Inthis case, the player tracking server 1000 may determine that anocclusion event has occurred between the first sport participant and thesecond sport participant.

As described above, it has been described that the player trackingserver 1000 may determine whether an occlusion event has occurred, butthe player tracking server 1000 may determine the severity of theocclusion event between the first sport participant and the second sportparticipant in additional consideration of a second preset thresholdvalue.

For example, FIG. 5C may be a diagram showing that the size orproportion of the region where the first bounding box BB1 and the secondbounding box BB2 overlap is smaller than the second preset thresholdvalue. In other words, in the case of FIG. 5C, the player trackingserver 1000 may determine that an occlusion event has occurred betweenthe first sport participant and the second sport participant but theseverity of the occlusion is not high.

On the contrary, FIG. 5D may be a diagram showing that the size orproportion of the region where the first bounding box BB1 and the secondbounding box BB2 overlap is greater than the first threshold value andthe second threshold value.

In this case, the player tracking server 1000 may determine that anocclusion event has occurred between the first sport participant and thesecond sport participant and the severity of the occlusion is high.

As another example, the player tracking server 1000 may detect anocclusion event on the basis of image data in the form of an RGB map (oran intensity map). Specifically, the player tracking server 1000 maydetect an occlusion event using pixel data included in an RGB map anddetermine whether the occlusion event has occurred.

For example, when the sum of the numbers of pixels corresponding to aplurality of players included in the current image data is decreased tobelow a certain percentage of the sum of the numbers of pixelscorresponding to a plurality of players included in image data when anocclusion event did not occur, the player tracking server 1000 may beimplemented to determine that an occlusion event has occurred. To thisend, any suitable threshold percentage may be preset for the certainpercentage.

In detail, referring to FIG. 6 , the player tracking server 1000 mayacquire pixel data related to the first sport participant and the secondsport participant from image data. In this case, the player trackingserver 1000 may detect or determine whether an occlusion event hasoccurred between the first sport participant and the second sportparticipant in consideration of pixel data related to the first sportparticipant and pixel data related to the second sport participant.

In detail, FIG. 6A is a diagram showing a plurality of pieces of pixeldata PD1 corresponding to the first sport participant and a plurality ofpieces of pixel data PD2 corresponding to the second sport participant.In this case, the plurality of pieces of pixel data PD1 corresponding tothe first sport participant and the plurality of pieces of pixel dataPD2 corresponding to the second sport participant may not overlap.

FIG. 6B is a diagram showing a plurality of pieces of pixel data PD3obtained by the plurality of pieces of pixel data PD1 corresponding tothe first sport participant and the plurality of pieces of pixel dataPD2 corresponding to the second sport participant partially overlapping.

According to a preferred embodiment, FIGS. 6A and 6B may show aplurality of pieces of pixel data related to the first sport participantand the second sport participant acquired from image data correspondingto two adjacent time points. In this case, the player tracking server1000 may detect or determine whether an occlusion event has occurred inconsideration of the ratio of the sum of the plurality of pieces ofpixel data PD3 of FIG. 6B to the sum of the plurality of pieces of pixeldata PD1 corresponding to the first sport participant and the pluralityof pieces of pixel data PD2 corresponding to the second sportparticipant of FIG. 6A.

For example, a reference threshold ratio value indicating whether anocclusion event has occurred may be preset for the ratio of the sum ofthe plurality of pieces of pixel data PD3 of FIG. 6B to the sum of theplurality of pieces of pixel data PD1 corresponding to the first sportparticipant and the plurality of pieces of pixel data PD2 correspondingto the second sport participant. In this case, the player trackingserver 1000 may determine that an occlusion event has occurred betweenthe first sport participant and the second sport participant when theratio of the sum of the plurality of pieces of pixel data PD3 of FIG. 6Bto the sum of the plurality of pieces of pixel data PD1 corresponding tothe first sport participant and the plurality of pieces of pixel dataPD2 corresponding to the second sport participant is smaller than thereference threshold ratio value.

However, the above description is just an example, and any suitablemethod using pixel data may be used to determine whether an occlusionevent has occurred. Also, by presetting an additional threshold ratiovalue, it is obvious that the player tracking server 1000 can determinenot only the occurrence of the occlusion event but also the severity ofthe occlusion event.

As another example, the player tracking server 1000 may detect anocclusion event on the basis of data regarding a player's locationincluded in positioning sensor data. Specifically, the player trackingserver 1000 may acquire data which is related to locations of aplurality of players and which is included in positioning sensor datacorresponding to the plurality of players. In this case, the playertracking server 1000 may determine whether an occlusion event hasoccurred in consideration ofthe number of players located within apredetermined radius from a specific player. Any suitable thresholdvalue may be preset for the number of players located within thepredetermined radius, and the player tracking server 1000 may determinethat an occlusion event has occurred when the number of players locatedwithin the predetermined radius from the specific player is greater thanthe threshold value. On the other hand, when the number of playerslocated within the predetermined radius from the specific player issmaller than the threshold value, the player tracking server 1000 maydetermine that no occlusion event has occurred.

The following description will refer to FIG. 7 . FIG. 7 is a diagramshowing locations of a plurality of sport participants.

The locations of the plurality of sport participants may be locationsacquired from image data or positioning sensor data. Preferably, thelocations of the plurality of sport participants may be locationsacquired from positioning sensor data. Also, preferably, the locationsof the plurality of sport participants may be locations acquired byperforming coordinate conversion on location-related data included inthe positioning sensor data. In this case, the player tracking server1000 may acquire the number of sport participants located in a region Rwithin a predetermined radius from a specific location L1_sensorcorresponding to a specific sport participant among locationscorresponding to the plurality of sport participants. Also, the playertracking server 1000 may determine whether an occlusion event hasoccurred in consideration of the number of sport participants located inthe region R within the predetermined radius from the specific locationL1_sensor. For example, a threshold value may be preset for the numberof sport participants located in the region R within the predeterminedradius, and the player tracking server 1000 may determine that anocclusion event has occurred when the number of sport participantslocated in the region R within the predetermined radius from thespecific location L1_sensor is greater than the threshold value. On theother hand, when the number of sport participants located in the regionR within the predetermined radius from the specific location L1_sensoris smaller than the threshold value, the player tracking server 1000 maydetermine that no occlusion event has occurred.

As another example, the player tracking server 1000 may acquire datarelated to the arrangement and orientation of the image capture device.Also, the player tracking server 1000 may receive positioning sensordata including data related to each player's location from a positioningsensor device 200 worn by the corresponding player. In this case, theplayer tracking server 1000 may detect whether an occlusion has occurredin consideration of positioning sensor data including the data relatedto players' locations and the data related to the arrangement andorientation of the image capture device. For example, the playertracking server 1000 may detect or determine whether an occlusion hasoccurred on the basis of the players' locations computed from thepositioning sensor data and data related to the line-of-sight and angleof the image capture device.

The following description will refer to FIG. 8 . FIG. 8 is a diagramshowing locations of a plurality of sport participants, a virtual point310 corresponding to the image capture device 300, and virtual lines 320and 322 for an orientation including the line-of-sight of the imagecapture device 300.

The locations of the plurality of sport participants may be locationsacquired from image data or positioning sensor data. According to apreferred example, locations L2_sensor and L3_sensor of the plurality ofsport participants may be locations acquired from the positioning sensordata. Also, preferably, the locations of the plurality of sportparticipants may be locations acquired by performing coordinateconversion on location-related data included in the positioning sensordata.

The virtual point 310 corresponding to the image capture device may be alocation acquired from arrangement information including locationinformation of the image capture device 300 located near the playfield.According to a preferred example, the virtual point 310 corresponding tothe image capture device may be a location acquired by performingcoordinate conversion on location-related data of the image capturedevice 300.

Information regarding the orientation of the image capture devicerelated to the virtual line 320 and information regarding theline-of-sight of the image capture device related to the virtual line322 may be acquired from the image capture device 300. According to apreferred example, the virtual line 322 corresponding to theline-of-sight of the image capture device may be a virtual linegenerated by performing coordinate conversion on the basis of a commoncoordinate system with the virtual point 310 or the locations L2_sensorand L3_sensor of the plurality of sport participants.

In this case, the player tracking server 1000 may detect or determinewhether an occlusion has occurred between a plurality of sportparticipants in consideration of the virtual line 322 corresponding tothe line-of-sight and the locations L2_sensor and L3_sensor of theplurality of sport participants. For example, the location L2_sensor ofa first sport participant may be acquired from positioning sensor dataacquired from positioning sensor devices 210 and 220 of the first sportparticipant, and the location L3_sensor of a second sport participantmay be acquired from positioning sensor data acquired from positioningsensor devices 210 and 220 of the second sport participant. In thiscase, when the location L2_sensor of the first sport participant and thelocation L3_sensor of the second sport participant are substantiallycollinear with respect to the line of sight, the player tracking server1000 may determine that an occlusion event has occurred between thefirst sport participant and the second sport participant.

As another example, the player tracking server 1000 may be implementedto determine whether an occlusion event has occurred in consideration ofwhether players related to the occlusion event are on the same team.Specifically, for the image data and especially image data in the formof an RGB map, teams of players may be distinguished on the basis of thesimilarity of pixels related to uniforms. Also, since identifiers ofindividual players are included in the positioning sensor data, teams ofplayers may be distinguished based on the identifiers.

In this case, when the players related to the occlusion event are on thesame team, the player tracking server 1000 may perform an operation ofdetermining that an occlusion event has occurred. On the other hand,when the players related to the occlusion event are not on the sameteam, the player tracking server 1000 may determine that no occlusionevent has occurred.

However, this is just an example, the player tracking server 1000 maydetermine that an occlusion event has occurred even though the playersrelated to the occlusion event are not on the same team and mayadditionally evaluate validity related to a player's location computedfrom the image data during an operation of determining the severity ofthe occlusion event.

As another example, the player tracking server 1000 may be provided todetect or determine whether an occlusion event has occurred inconsideration of a situation during a sport game. Specifically, during asporting game, there may be a situation in which a plurality of playersare concentrated, especially a corner kick situation. In this case, theplayer tracking server 1000 may recognize such a dense situation toestimate that an occlusion event has occurred.

The above-described occlusion event detection methods are just examples,and it is obvious that any suitable method for detecting an occlusionevent between a plurality of players may be provided to the playertracking server 1000 in consideration of a player's location computedfrom image data and positioning sensor data and in an image in the formof an intensity map or an RGB map.

As described above, the player tracking server 1000 according to anembodiment of the present disclosure may detect whether an occlusionevent has occurred between a plurality of players in the image data, andwhen the player tracking server 1000 determines that an occlusion eventhas occurred, a player's location computed from the image data may bedetermined to be invalid.

The player tracking server 1000 according to an embodiment of thepresent disclosure may be provided to evaluate the validity of theplayer's location computed from the image data in further considerationof the severity of the occlusion event between the plurality of playersin the image data. Here, the severity of the occlusion event may be usedin the sense of encompassing a concept of digitizing or quantifying theseverity of the occlusion event in which the plurality of players atleast partially overlap.

As an example, referring to FIG. 8 again, the player tracking server1000 according to an embodiment of the present disclosure may determinethe severity of the occlusion event in consideration of the number ofsport participants located in the region R within the predeterminedradius from the specific location L1_sensor and thus may evaluate thevalidity of the player's location computed from the image data.

For example, a plurality of threshold values (a first threshold valueand a second threshold value) may be preset for the number of sportparticipants located in the region R within the predetermined radius,and the player tracking server 1000 may determine that the occlusionevent has occurred and that the severity of the occlusion event is low(mild) when the number of sport participants located in the region Rwithin the predetermined radius from the specific location L1_sensor isgreater than the first threshold value and smaller than the secondthreshold value.

On the other hand, when the number of sport participants located in theregion R within the predetermined radius from the specific locationL1_sensor is greater than the second threshold value, the playertracking server 1000 may determine that an occlusion event has occurredand that the severity of the occlusion event is high (severe).

Apart from the severity of the occlusion event, when the number of sportparticipants located in the region R within the predetermined radiusfrom the specific location L1_sensor is smaller than the first thresholdvalue, the player tracking server 1000 may determine that no occlusionevent has occurred.

As another example, the player tracking server 1000 according to anembodiment of the present disclosure may be provided to evaluate thevalidity of the player's location computed from the image data infurther consideration of a team identifier even though it is determinedthat the occlusion event has occurred. For example, even though anocclusion event of a plurality of sport participants included in theimage data has occurred, the player tracking server 1000 may evaluatethe validity of the player's location computed from the image data byadditionally determining the severity of the occlusion event accordingto a team identifier of the players related to the occlusion event.

Generally, when the sport participants are on the same team, theuniforms of the sport participants are the same. Thus, when an occlusionevent occurs between the sport participants belonging to the same team,it is highly likely that the locations of the sport participantscomputed from the image data are invalid.

On the other hand, when the sport participants are on different teams,the uniforms of the sport participants are different. Thus, even when anocclusion event has occurred between the sport participants belonging todifferent teams, the locations of the sport participants may be validlyacquired using pixel data of an RGB map associated with the color of theuniforms.

Therefore, the player tracking server 1000 according to an embodiment ofthe present disclosure may evaluate the validity of the player'slocation computed from the image data in further consideration of a teamidentifier even though it is determined that the occlusion event hasoccurred.

Pixel data related to the color of the uniforms acquired from the imagedata is illustrated as the team identifier, but this is just an example,and the team identifier, as well as the player identifier, may befurther included in the positioning sensor data.

The following description will refer to FIG. 9 . FIGS. 9A and 9B areexemplary diagrams of a method of determining the validity of theplayer's location computed from the image data according to the presentdisclosure.

As an example, the player tracking server 1000 may be provided toevaluate the validity of the player's location computed from the imagedata in consideration of the player's vertical movement included in theimage data. Here, the vertical movement may have a meaning encompassingall movements in a direction perpendicular to a coordinate system planerelated to the player's location computed from the image data.Specifically, data related to the vertical movement may be furtheracquired from an inertial sensor worn by players, and when the verticalmovement is present, there is a possibility that an error may occur inthe player's location computed from the image data. Therefore, theplayer tracking server 1000 may evaluate the validity of the player'slocation computed from the image data in consideration of the player'svertical movement included in the image data.

Referring to FIG. 9A, when there is substantially no vertical movementof a sport participant 1 included in image data, the difference betweena location L1_image of the sport participant 1 computed from the imagedata and an actual location L1 of the sport participant 1 may berelatively small. In other words, when there is substantially novertical movement of the sport participant 1 included in the image data,the location L1_image of the sport participant 1 computed from the imagedata may approximate the actual location L1.

On the other hand, referring to FIG. 9B, when there is a verticalmovement of the sport participant 1 included in the image data, thedifference between a location L2_image of the sport participant 1computed from the image data and an actual location L2 of the sportparticipant 1 may be relatively large. In other words, when a verticalmovement of the sport participant 1 is included in the image data, thelocation L2_image of the sport participant 1 computed from the imagedata is likely to have a significant error with respect to the actuallocation L2.

Therefore, the player tracking server 1000 may evaluate the validity ofthe player's location computed from the image data in consideration ofthe player's vertical movement included in the image data. For example,when a player's vertical movement included in the image data isdetected, that is, when a change in z-direction data is detected by aninertial sensor wom by the sport participant 1, the player trackingserver 1000 may determine that the location of the sport participant 1computed from the image data is invalid.

As another example, when a player's vertical movement included in theimage data is not detected, that is, when a change in z-direction datais not detected by an inertial sensor worn by the sport participant 1,the player tracking server 1000 may determine that the location of thesport participant 1 computed from the image data is valid.

As described above, it has been described that it is possible toevaluate the validity of a sport participant computed from the imagedata depending on whether the vertical movement of the sport participantis detected. This is just an example, and even when a vertical movementis detected, the validity of the location of the sport participantcomputed from the image data may be determined in further considerationof the direction or magnitude of acceleration in the z-direction. Also,information regarding the height, location, or capture angle of theimage capture device 300 may be additionally provided and considered indetermining whether the location of the sport participant computed fromthe image data is verified.

The following description will refer to FIG. 10 . FIG. 10 is anexemplary diagram of a method of determining the validity of a player'slocation computed from image data according to the present disclosure.

As an example, the player tracking server 1000 may evaluate the validityof the player's location computed from the image data in considerationof an internal disparity of the player's location computed from theimage data. Specifically, the player tracking server 1000 maycontinuously compute the player's location from the image data. In thiscase, when a change in the computed location of the player is greaterthan a predetermined threshold value, the player tracking server 1000may determine that data regarding the greatly changed location of theplayer is invalid. In other words, when the change between the player'slocation computed from image data corresponding to a first time pointand the player's location computed from image data corresponding to asecond time point is greater than the predetermined threshold value, theplayer tracking server 1000 may determine that the player's locationcomputed from the image data corresponding to the second time point isinvalid.

Referring to FIG. 10 , the player tracking server 1000 may be providedto compute or acquire locations of a specific sport participant fromimage data over time. In this case, when the difference between alocation Lt1_image of the sport participant computed from the first timepoint and a location Lt2_image of the sport participant computed fromthe second time point is greater than the predetermined threshold value,the player tracking server 1000 may determine that the locationLt2_image of the sport participant computed from the second time pointis invalid. On the other hand, when the difference between a locationLt3_image of the sport participant computed from a third time point anda location Lt4_image of the sport participant computed from a fourthtime point is smaller than the predetermined threshold value, the playertracking server 1000 may determine that the location Lt4_image of thesport participant computed from the fourth time point is valid.

As described above, it has been described that the validity of thelocation of the sport participant computed from the image data over timeis determined based on the preset threshold value, but this is just anexample. Any suitable method may be used to determine the validity ofthe location of the sport participant computed from the image data. Forexample, the player tracking server 1000 may compute a trend line or thelike by quantifying the tendency of the locations of the sportparticipant over time and may determine the validity of the location ofthe sport participant computed from the image data on the basis of themagnitude of deviation of the trend line or the like.

Also, the player tracking system 100 according to an embodiment of thepresent disclosure may determine the validity of an acquired positioningsensor-based location. Specifically, the player tracking server 1000 mayevaluate the validity of the player's location computed from thepositioning sensor data. More specifically, the player tracking server1000 may consider at least one of reliability-related information of thepositioning sensor data and the internal disparity of the player'slocation computed from the positioning sensor data in order to evaluatethe validity of the player's location computed from the positioningsensor data.

The following description will refer to FIG. 11 . FIG. 11 is anexemplary diagram of a method of determining the validity of a player'slocation calculated from positioning sensor data according to thepresent disclosure.

As an example, the player tracking server 1000 may evaluate the validityof the player's location computed from the positioning sensor data inconsideration of reliability-related information of a sensor signalincluded in the positioning sensor data.

For example, the reliability-related information (e.g., DoP, SNR) of asignal may be included in a signal transmitted from the GlobalNavigation Satellite System (GNSS) to a positioning sensor device.

In this case, the player tracking server 1000 may acquire thereliability-related information through the communication module 1100,and the player tracking server 1000 may evaluate the validity of theplayer's location computed from the positioning sensor data inconsideration of the acquired reliability-related information of thesensor signal.

For example, a threshold value may be preset for the reliability-relatedinformation, and when the reliability-related information included inthe positioning sensor data is greater than the preset threshold value,the player tracking server 1000 may determine that the player's locationcomputed from the positioning sensor data is valid. On the other hand,when the reliability-related information included in the positioningsensor data is smaller than the preset threshold value, the playertracking server 1000 may determine that the player's location computedfrom the positioning sensor data is invalid.

Referring to FIG. 11 , the reliability-related information of the sensorsignal may be different for each region of the playfield. For example,when a sport participant is located in a first region A1 of theplayfield, the reliability of the sensor signal may be relatively low.Therefore, location data included in positioning sensor data acquiredfrom a positioning sensor device of the sport participant located in thefirst region A1 may have relatively low reliability. On the other hand,when a sport participant is located in a second region A2 of theplayfield, the reliability of the sensor signal may be relatively high.Therefore, location data included in positioning sensor data acquiredfrom a positioning sensor device of the sport participant located in thesecond region A2 may have relatively high reliability.

In this case, when the reliability of the sensor signal in the firstregion A1 is higher than the predetermined threshold value, the playertracking server 1000 may determine that the location of the sportparticipant computed from the positioning sensor data acquired from thepositioning sensor device of the sport participant located in the firstregion A1 is valid. On the other hand, when the reliability of thesensor signal in the second region A2 is lower than the predeterminedthreshold value, the player tracking server 1000 may determine that thelocation of the sport participant computed from the positioning sensordata acquired from the positioning sensor device of the sportparticipant located in the second region A2 is invalid.

Also, the player tracking server 1000 may determine that the location ofthe sport participant computed based on data with high reliability isvalid by quantifying and comparing the reliability of the sensor signaland the reliability of image data or may compute the location of thesport participant through weight values corresponding to the reliabilityof the sensor signal and the reliability of the image data. This will bedescribed below with reference to FIG. 22 .

In FIG. 11 , it has been described that the first region and the secondregion are limited to specific regions in the playfield, but this isjust an example for describing that the reliability of the sensor signalis different for each region.

The following description will refer to FIG. 12 . FIG. 12 is anexemplary diagram illustrating a method of determining the validity of aplayer's location computed from positioning sensor data according to thepresent disclosure.

As an example, the player tracking server 1000 may evaluate the validityof the player's location computed from the positioning sensor data inconsideration of an internal disparity of the player's location computedfrom the positioning sensor data. Specifically, the player trackingserver 1000 may continuously compute or acquire the player's locationfrom the positioning sensor data. In this case, when a change in thecomputed or acquired location of the player is greater than apredetermined threshold value, the player tracking server 1000 maydetermine that data regarding the greatly changed location of the playeris invalid. In other words, when the change between the player'slocation computed from positioning sensor data corresponding to a firsttime point and the player's location computed from positioning sensordata corresponding to a second time point is greater than thepredetermined threshold value, the player tracking server 1000 maydetermine that the player's location computed from the positioningsensor data corresponding to the second time point is invalid.

Referring to FIG. 12 , the player tracking server 1000 may compute oracquire locations of a specific sport participant from positioningsensor data over time. In this case, when the difference between alocation Lt1_sensor of the sport participant computed from the firsttime point and a location Lt2_sensor of the sport participant computedfrom the second time point is greater than the predetermined thresholdvalue, the player tracking server 1000 may determine that the locationLt2_sensor of the sport participant computed from the second time pointis invalid. On the other hand, when the difference between a locationLt3_sensor of the sport participant computed from a third time point anda location Lt4_sensor of the sport participant computed from a fourthtime point is smaller than the predetermined threshold value, the playertracking server 1000 may determine that the location Lt4_sensor of thesport participant computed from the fourth time point is valid.

As described above, it has been described that the validity of thelocation of the sport participant computed from the positioning sensordata over time is determined based on the preset threshold value, butthis is just an example. Any suitable method may be used to determinethe validity of the location of the sport participant computed from thepositioning sensor data. For example, the player tracking server 1000may compute a trend line or the like by quantifying the tendency of thelocations of the sport participant over time and may determine thevalidity of the location of the sport participant computed from thepositioning sensor data on the basis of the magnitude of deviation ofthe trend line or the like.

The player tracking system 100 according to an embodiment of the presentdisclosure may determine the validity of at least one of an image-basedlocation and a positioning sensor-based location on the basis of thedifference between the image-based location and the positioningsensor-based location. Specifically, the player tracking server 1000 maybe provided to determine the validity of at least one of the image-basedlocation and the positioning sensor-based location. More specifically,the player tracking server 1000 may be provided to determine thevalidity of at least one of the image-based location and the positioningsensor-based location on the basis of an external disparity between theimage-based location and the positioning sensor-based location.

As an example, the player tracking server 1000 may continuously computeor acquire a player's location from each of positioning sensor data andimage data. In this case, the player tracking server 1000 may evaluatethe validity of at least one of the player's location computed from theimage data and the player's location computed from the positioningsensor data on the basis of a disparity index considering the difference(hereinafter referred to as an external disparity) between a firstlocation of the player computed from the image data and a secondlocation of the player computed from the positioning sensor data at thesame time point.

For example, a disparity index between the first location and the secondlocation computed at a first time point may be a first disparity index,and a disparity index between the first location and the second locationcomputed at a second time point temporally adjacent to the first timepoint may be a second disparity index.

In this case, when the absolute value of the first disparity index ΔD1is less than a predetermined first threshold value, the player trackingserver 1000 may determine that at least one of the first location of theplayer computed from the image data at the first time point and thesecond location of the player computed from the positioning sensor dataat the first time point is valid.

On the other hand, when the absolute value of the second disparity indexΔD2 exceeds a predetermined second threshold value, the player trackingserver 1000 may evaluate that at least one of the first location of theplayer computed from the image data at the second time point and thesecond location of the player computed from the positioning sensor dataat the second time point is invalid.

Here, the first threshold value and the second threshold value may bethe same or may be preset to any suitable values different from eachother.

As another example, the player tracking server 1000 may evaluate thevalidity of at least one of the player's location computed from theimage data and the player's location computed from the positioningsensor data in consideration of the difference |ΔD1|-|ΔD2| or |ΔD1-ΔD2|between the first disparity index ΔD1 and the second disparity indexΔD2.

Specifically, when the difference |ΔD1|-|ΔD2| or |ΔD1-ΔD2| between thefirst disparity index ΔD1 and the second disparity index ΔD2 is lessthan a third predetermined threshold value (preferably, when it isdetermined that the first location and the second location which arerelated to the first disparity index are valid), that is, when thedifference between the first location of the player computed from theimage data at the second time point and the second location of theplayer computed from the positioning sensor data at the second timepoint can be regarded as being within an error range, the playertracking server 1000 may determine that the first location and the firstlocation at the second time point are valid.

On the other hand, when the difference |ΔD1|-|ΔD2| or |ΔD1-ΔD2| betweenthe first disparity index ΔD1 and the second disparity index ΔD2 exceedsa fourth predetermined threshold value (preferably, when it isdetermined that the first location and the second location which arerelated to the first disparity index are valid), that is, when a timepoint at which the difference between the first location of the playercomputed from the image data at the second time point and the secondlocation of the player computed from the positioning sensor data at thesecond time point changes significantly is detected, the player trackingserver 1000 may determine that at least one of the first location andthe second location at the second time point is invalid.

Here, the third threshold value and the fourth threshold value may bethe same or may be preset to any suitable values different from eachother.

The player tracking system 100 according to an embodiment of the presentdisclosure may be provided to generate reliability indices related to animage-based location and a positioning sensor-based location.Specifically, the player tracking server 1000 may be provided togenerate the reliability index related to the image-based location andthe reliability index related to the positioning sensor-based locationon the basis of a validity determination result for the image-basedlocation and the positioning sensor-based location.

Specifically, the player tracking server 1000 may generate a firstreliability index of the player's location computed from positioningsensor data on the basis of the result of evaluating the validity of thepositioning sensor data. For example, the first reliability index may bequantified from reliability-related information of a signal included inthe positioning sensor data.

Also, the player tracking server 1000 may generate a second reliabilityindex of the player's location computed from image data on the basis ofthe result of evaluating the validity of the image data. In this case,the second reliability index may be quantified as an index correspondingto the first reliability index so that the second reliability index canbe quantitatively compared to the first reliability index.

In this case, the first reliability index and second reliability indexmay be used as one factor that should be considered in finallydetermining the location of the sport participant, which will bedescribed below.

The player tracking system 100 according to an embodiment of the presentdisclosure may be provided to convert coordinates between theimage-based location and the positioning sensor-based location.Specifically, the player tracking server 1000 may perform coordinateconversion between an image-based location and a positioningsensor-based location. Specifically, the player tracking server 1000 maybe provided to perform coordinate conversion between an image-basedlocation and a positioning sensor-based location. Thus, the image-basedlocation may have the same coordinates as the positioning sensor-basedlocation.

Specifically, the location of the player computed from the image datamay be computed for a first coordinate system, and the location of theplayer computed from the positioning sensor data may be computed for asecond coordinate system. In this case, the location of the playershould be determined in consideration of both of the location of theplayer computed from the image data and the location of the playercomputed from the positioning sensor data or the location of the playerif it is necessary to track the player, or the location of the playershould be determined for a common coordinate system when switching isperformed between the location of the player computed from the imagedata and the location of the player computed from the positioning sensordata. Thus, a conversion operation should be performed between the firstcoordinate system and the second coordinate system.

As an example, the second coordinate system may be converted to thefirst coordinate system.

As another example, the first coordinate system may be converted to thesecond coordinate system.

As still another example, the computed location of the player may beconverted from the first coordinate system or the second coordinatesystem to an absolute coordinate system different from the firstcoordinate system and the second coordinate system.

However, the above-described conversion between the coordinate systemsis just an example, and a player's location computed from image data anda player's location computed from positioning sensor data may bevariously converted to a common coordinate system according to anysuitable method.

In this case, the player tracking server 1000 may be provided to usevarious coordinate conversion algorithms.

As an example, the player tracking server 1000 may be implemented usingan artificial neural network that is trained by any suitable method suchas machine learning or deep learning.

The following description will refer to FIGS. 13 and 14 .

FIG. 13 is a diagram illustrating that an artificial neural network istrained to convert a location in a first coordinate system related topositioning sensor data to a location in a second coordinate systemrelated to image data.

The following description will refer to FIG. 14 . FIG. 14 is a diagramillustrating that a location in the first coordinate system related topositioning sensor data is converted to a location in the secondcoordinate system related to image data using the trained artificialneural network.

Referring to FIG. 14 , the player tracking server 1000 according to anembodiment of the present disclosure may perform coordinate conversionfrom a location in a first coordinate system related to positioningsensor data to a location in a second coordinate system related to imagedata using an artificial neural network.

For example, when it is determined that an image-based location acquiredfrom image data is invalid, there is a need to determine a location of asport participant from a positioning sensor-based location acquired frompositioning sensor data. However, since the positioning sensor-basedlocation and the image-based location may have different coordinatesystems, the player tracking server 1000 may convert an acquiredpositioning sensor-based location of the first coordinate system to alocation of the second coordinate system related to image data. Here,the image-based location being invalid may refer to at least one of acase in which an occlusion is detected between sport participants and itis determined that an occlusion event is severe, a case in which avertical movement of a sport participant is detected, and a case inwhich an internal disparity between image-based locations is present.

As an example, an artificial intelligence network to be trained toconvert a location in the first coordinate system related to positioningsensor data to a location in the second coordinate system related toimage data may include an input layer, a hidden layer including aplurality of nodes, and an output layer.

In order to train the artificial neural network, an image-based location(x_image, y_image) determined to be valid and a positioning sensor-basedlocation (x_sensor, y_sensor) corresponding to the image-based locationdetermined to be valid may be input to the input layer. Also, in orderto perform accurate training, a positioning sensor-based speed|v|_sensor may be additionally input to the input layer in addition tothe positioning sensor-based location (x_sensor, y_sensor) correspondingto the image-based location determined to be valid. In this case, thepositioning sensor-based speed |v|_sensor may be computed from thepositioning sensor-based location (x_sensor, y_sensor). Alternatively,the positioning sensor-based speed |v|_sensor may be computed from dataacquired from an accelerometer of an inertial sensor worn by a sportsensor.

For example, referring to FIG. 13 , the image data may include imagedata (image 1) in which no occlusion is detected, image data (image 2)in which an occlusion is detected but is determined not to be severe,and image data (image 3) in which an occlusion is detected and isdetermined to be severe.

In this case, as described above, an image-based location acquired bythe image data (image 1) in which no occlusion is detected may be valid.That is, the image data (image 1) may be utilized to train theartificial neural network. Therefore, a first image-based location(x_image1, y_image1) acquired by the image data (image 1) in which noocclusion is detected and a first positioning sensor-based location(x_sensor1, y_sensor1) and a first positioning sensor-based speed|v|_sensor1 corresponding to the first image-based location may be usedas input values for a training set.

In this case, as described above, an image-based location acquired bythe image data (image 3) in which an occlusion is detected and isdetermined to be severe may be invalid. That is, when the image data(image 3) is utilized, to train the artificial neural network, theaccuracy of the training may be reduced. Therefore, data related to theimage data (image 3 in which an occlusion is detected and is determinedto be severe may not be used as a training set.

Also, the image data (image 2) in which an occlusion is detected but isdetermined not to be severe may be appropriately selected to be used ornot to be used as a training set in consideration of the above-describedcriterion.

Referring to FIG. 13 again, the positioning sensor-based location(x_sensor, y_sensor) and the positioning sensor-based speed |v|_sensorcorrespond to the image-based location that is input to the input layerand that is determined to be valid may be converted to a location(x′_image, y′_image) in coordinates of the image-based location throughthe hidden layer as an output value of the output layer.

In this case, the output value (x′_image, y′_image) may be compared tothe image-based location (x_image, y_image) input to the input layer.Thus, by adjusting a weight value of a node included in the hidden layeron the basis of a difference between the output value and theimage-based location, the artificial neural network may be trained.

Referring to FIG. 14 , a location in the first coordinate system relatedto positioning sensor data may be converted to a location in the secondcoordinate system related to image data using the trained artificialneural network of FIG. 13 . Preferably, the trained artificial neuralnetwork according to an embodiment of the present disclosure may be usedto convert a location in the first coordinate system related topositioning sensor data to a location in the second coordinate systemrelated to image data when it is determined that an image-based locationis invalid.

For example, the image data may include image data (image 1) in which noocclusion is detected, image data (image 2) in which an occlusion isdetected but is determined not to be severe, and image data (image 3) inwhich an occlusion is detected and is determined to be severe.

In this case, among the pieces of image data, an image-based location(x_image3, y_image3) acquired by the image data (image 3) in which anocclusion is detected and is determined to be severe may be determinedto be invalid. In this case, when a positioning sensor-based location(x_sensor3, y_sensor3) and a positioning sensor-based speed |v|_sensor3corresponding to the image-based location (x_image3, y_image3) acquiredby the image data (image 3) in which an occlusion is detected and isdetermined to be severe are input to the input layer of the trainedartificial neural network, the positioning sensor-based location(x_sensor3, y_sensor3) may be converted to a coordinate system relatedto the image-based location. Thus, even when the image-based location isinvalid, the location of the sport participant may be continuouslyacquired based on the positioning sensor-based location.

In this case, among the pieces of image data, with regard to the imagedata (image 2) in which an occlusion is detected but is determined notto be severe, a reliability index of the image-based location may becompared to a reliability index of the positioning sensor-basedlocation, which will be described below. When the positioningsensor-based location has higher reliability than the image-basedlocation, the positioning sensor-based location may be provided toconvert a coordinate value using the trained artificial neural network.

As described above, it has mainly been described that the positioningsensor-based location of the first coordinate system is converted to alocation of the second coordinate system related to the image-basedlocation. However, the artificial neural network may be trained toconvert the positioning sensor-based location to a location in anysuitable common coordinate system.

Also, it has mainly been described that the coordinates of thepositioning sensor-based location are converted when the image-basedlocation is invalid, but this is just an example. When it is determinedthat the positioning sensor-based location is invalid, it is obviousthat it is possible to train or deploy an artificial neural network thatconverts the image-based location to the positioning sensor-basedlocation in a similar method.

As described above, here, the player tracking server 1000 may usevarious coordinate conversion algorithms for coordinate conversion.

As an example, the coordinate conversion algorithm may be provided as amachine learning model. A representative example of the machine learningmodel may be an artificial neural network. Specifically, arepresentative example of the artificial neural network is adeep-learning-based artificial neural network including an input layerthat receives data, an output layer that outputs a result, and a hiddenlayer that is between the input layer and the output layer to processdata. Detailed examples of the artificial neural network are aconvolutional neural network, a recurrent neural network, a deep neuralnetwork, and the like. Here, the artificial neural network should beinterpreted in a comprehensive sense including the above-describedartificial neural networks, other various types of artificial neuralnetworks, and all combinations of the artificial neural networks, anddoes not necessarily have to be based on deep learning.

In addition, the machine learning model does not necessarily have to bein the form of an artificial neural network model. In addition, themachine learning model may include the k-nearest neighbors (KNN)algorithm, Random Forest, support vector machine (SVM), principalcomponent analysis (PCA), and the like and may include ensembles of theaforementioned techniques or other various combinations thereof.Meanwhile, it should be noted that the artificial neural network can bereplaced with another machine learning model unless otherwise stated inthe embodiments mentioned focusing on the artificial neural network.

Furthermore, the coordinate conversion algorithm herein is notnecessarily limited to the machine learning model. That is, thecoordinate conversion algorithm may include various decision algorithmsother than the machine learning model.

Therefore, it should be noted that the coordinate conversion algorithmherein should be understood in a comprehensive sense including all typesof algorithms that perform coordinate conversion using data in a firstcoordinate system.

The player tracking system 100 according to an embodiment of the presentdisclosure may be provided to perform interpolation upon coordinateconversion between the image-based location and the positioningsensor-based location. Specifically, the player tracking server 1000 mayadditionally perform an operation of interpolating a convertedcoordinate value in order to correct an error that may occur during acoordinate conversion between the player's location computed from imagedata and the player's location computed from positioning sensor data.For example, while tracking the player using the player's locationcomputed from the image data, the player tracking server 1000 may trackthe player using the player's location computed from positioning sensordata as necessary when it is determined that the player's locationcomputed from the image data is invalid. In this case, when the player'slocation computed from the positioning sensor data is convertedaccording to a first coordinate system of the player's location computedfrom the image data, there may be an error in the converted locationvalue.

Therefore, the player tracking server 1000 according to an embodiment ofthe present disclosure may perform an operation of interpolating a valueobtained by performing coordinate conversion on the player's locationcomputed from the positioning sensor data according to the firstcoordinate system using the player's location which has been trackedbefore and which is computed from the image data.

The player tracking system 100 according to an embodiment of the presentdisclosure may determine the player's location in consideration of animage-based location and a positioning sensor-based location.

Specifically, the player tracking server 1000 may be provided to finallydetermine the player's location on the basis of an image-based locationand a positioning sensor-based location.

As an example, when the image-based location is valid, the playertracking server 1000 may be provided to determine the player's locationon the basis of the player's location computed from the image data.

On the other hand, when the image-based location is invalid, the playertracking server 1000 may determine the player's location on the basis ofthe location computed from the positioning sensor data.

For example, when it is determined or detected that there is noocclusion event between a plurality of players included in the imagedata and thus it is determined that the player's location computed fromthe image data is valid, the player tracking server 1000 may determinethe player's location on the basis of the player's location computedfrom the image data. On the other hand, when it is determined ordetected that there is an occlusion event between a plurality of playersincluded in the image data and thus it is determined that the player'slocation computed from the image data is invalid, the player trackingserver 1000 may determine the player's location on the basis of thelocation computed from the positioning sensor data.

As another example, when no vertical movement is detected with regard toa player included in the image data and thus it is determined that theplayer's location computed from the image data is valid, the playertracking server 1000 may determine the player's location on the basis ofthe player's location computed from the image data. On the other hand,when a vertical movement is detected with regard to a player included inthe image data and thus it is determined that the player's locationcomputed from the image data is invalid, the player tracking server 1000may determine the player's location on the basis of the locationcomputed from the positioning sensor data.

As another example, when an internal disparity between the player'slocations computed from the image data does not exceed a predeterminedthreshold value and thus it is determined the player's location computedfrom the image data is valid, the player tracking server 1000 maydetermine the player's location on the basis of the player's locationcomputed from the image data. On the other hand, when an internaldisparity between the player's locations computed from the image dataexceeds the predetermined threshold value and thus it is determined theplayer's location computed from the image data is invalid, the playertracking server 1000 may determine the player's location on the basis ofthe location computed from the positioning sensor data.

In an embodiment, when the player's location computed from thepositioning sensor data is valid, the player tracking server 1000 maydetermine the player's location on the basis of the player's locationcomputed from the positioning sensor data. On the other hand, when theplayer's location computed from the positioning sensor data is invalid,the player tracking server 1000 may determine the player's location onthe basis of the location computed from the image data.

As an example, the player tracking server 1000 may determine theplayer's location on the basis of the determination that the player'slocation computed from the positioning sensor data is valid based on thereliability-related information of the signal included in thepositioning sensor data.

For example, when it is determined that the player's location computedfrom the positioning sensor data is valid because the reliability of thesignal included in the positioning sensor data is greater than a presetthreshold value, the player tracking server 1000 may determine theplayer's location on the basis of the player's location computed fromthe positioning sensor data.

On the other hand, when it is determined that the player's locationcomputed from the positioning sensor data is invalid because thereliability of the signal included in the positioning sensor data issmaller than the preset threshold value, the player tracking server 1000may determine the player's location on the basis of the player'slocation computed from the image data.

As another example, when an internal disparity between the player'slocations computed from the positioning sensor data does not exceed apredetermined threshold value and thus it is determined the player'slocation computed from the positioning sensor data is valid, the playertracking server 1000 may determine the player's location on the basis ofthe player's location computed from the positioning sensor data.

On the other hand, when an internal disparity between the player'slocations computed from the positioning sensor data exceeds thepredetermined threshold value and thus it is determined the player'slocation computed from the positioning sensor data is invalid, theplayer tracking server 1000 may determine the player's location on thebasis of the location computed from the image data.

In an embodiment, the player tracking server 1000 may finally determinethe player's location on the basis of the disparity indices of or thedifference between the player's location computed from the image dataand the player's location computed from the positioning sensor data.

Specifically, as described above, on the basis of the first disparityindex ΔD1 between the first location of the player computed from theimage data at the first time point and the second location of the playercomputed from the positioning sensor data at the first time point andthe second disparity index ΔD2 between the first location of the playercomputed from the image data at the second time point and the secondlocation of the player computed from the positioning sensor data at thesecond time point, it may be determined that at least one of the firstlocation and the second location is invalid. In this case, the playertracking server 1000 may finally determine the player's location on thebasis of the determination of the validity based on the disparityindices.

For example, when a computed disparity index is smaller than apredetermined first threshold value, that is, when it can be determinedthat both of the player's location computed from the positioning sensordata and the player's location computed from the image data are valid,the player tracking server 1000 may determine the player's location onthe basis of one of the player's location computed from the positioningsensor data and the player's location computed from the image data.

On the other hand, when a computed disparity index is greater than apredetermined second threshold value, that is, when it can be determinedthat at least one of the player's location computed from the positioningsensor data and the player's location computed from the image data isinvalid, the player tracking server 1000 may determine the player'slocation on the basis of the one of the player's location computed fromthe positioning sensor data and the player's location computed from theimage data that has a higher reliability index.

The following description will refer to FIG. 15 . FIG. 15 is a diagramillustrating an exemplary method of determining a player's location onthe basis of a reliability index according to an embodiment of thepresent disclosure.

In an embodiment, the player tracking server 1000 may determine theplayer's location on the basis of a comparison between the firstreliability index related to the player's location computed from theimage data and the second reliability index related to the player'slocation computed from the positioning sensor data.

As an example, when the first reliability index is higher than thesecond reliability index (FIG. 15A or FIG. 15B), the player trackingserver 1000 may determine the player's location on the basis of theplayer's location computed from the image data.

For example, as shown in FIG. 15A, when the first reliability index ishigher than a predetermined reliability threshold value and the secondreliability index is lower than the predetermined reliability thresholdvalue, the player tracking server 1000 may determine the location of thesport participant on the basis of the location of the sport participantcomputed from image data.

For example, as shown in FIG. 15B, when the first reliability index ishigher than the second reliability index and the first reliability indexand the second reliability index are higher than the predeterminedreliability threshold value, the player tracking server 1000 maydetermine the location of the sport participant on the basis of thelocation of the sport participant computed from the image data, similarto FIG. 15A.

As another example, when the second reliability index is higher than thefirst reliability index (FIG. 15C), the player tracking server 1000 maydetermine the player's location on the basis of the player's locationcomputed from the positioning sensor data.

As another example, as shown in FIG. 15C, when the second reliabilityindex is higher than the predetermined reliability threshold value andthe first reliability index is lower than the predetermined reliabilitythreshold value, the player tracking server 1000 may determine thelocation of the sport participant on the basis of the location of thesport participant computed from positioning sensor data.

As another example, the player tracking server 1000 may determine theplayer's location by assigning weight values in consideration of thefirst reliability index related to the first location of the playercomputed from the image data and the second reliability index related tothe second location of the player computed from the positioning sensordata. Specifically, the player tracking server 1000 may determine theplayer's location by assigning weight values considering the firstreliability index and the second reliability index to the first locationand the second location, respectively.

For example, as shown in FIG. 15B, when both of the first reliabilityindex and the second reliability index are higher than the predeterminedreliability threshold value, the player tracking server 1000 maydetermine the location of the sport participant by assigning weightvalues corresponding to the reliability indices to the first location ofthe sport participant computed from the image data and the secondlocation of the sport participant computed from the positioning sensordata.

As another example, as shown in FIG. 15D, when the first reliabilityindex is similar to or substantially the same as the second reliabilityindex, the player tracking server 1000 may determine the location of thesport participant by assigning weight values corresponding to thereliability indices to the location of the sport participant computedfrom the image data and the location of the sport participant computedfrom the positioning sensor data.

As described above, it has been described that the location of the sportparticipant may be determined by assigning weight values correspondingto reliability indices in the case of FIGS. 15B and 15D, but this isjust an example. It is obvious that the location of the sportparticipant may be determined by appropriately assigning the weightvalues corresponding to the reliability indices in the case of FIG. 15Aor FIG. 15C or in any case.

The location of the sport participant determined by the player trackingserver 1000 according to an embodiment of the present disclosure may betransferred to the memory 1200 through the communication module 1100 andstored in the memory 1200. Alternatively, the location of the sportparticipant determined by the player tracking server 1000 may betransmitted to an external server through the communication module 1100or uploaded to the Internet.

The configuration and operation of the player tracking server 1000according to an embodiment of the present disclosure have been describedabove. A player tracking method according to this embodiment will bedescribed below. In the following description, the player trackingmethod according to an embodiment of the present disclosure is performedby the above-described player tracking system 100. However, this is justfor convenience of description, and thus the player tracking methodaccording to an embodiment of the present disclosure is not limited tothe above-described player tracking system 100. That is, the playertracking method, which will be described below, does not necessarilyhave to be performed only by the player tracking system 100 but may beperformed by another system or device having a function similar to thatof the above-described player tracking system 100.

In addition, the configuration and operation of the player trackingserver 1000 according to an embodiment of the present disclosure may beappropriately applied to other embodiments of the player tracking system100, which will be described below.

The following description will refer to FIG. 16 . FIG. 16 is a flowchartillustrating a player tracking method implemented by the player trackingsystem 100 according to an embodiment of the present disclosure.

The player tracking method according to an embodiment of the presentdisclosure may include acquiring image data (S1100), acquiringpositioning sensor data (S1200), and determining a player's location(S1300).

In the operation of acquiring the image data (S1100), image dataacquired from an image capture device 300 located near a playfield maybe acquired.

In the operation of acquiring the positioning sensor data (S1200),positioning sensor data transmitted from a positioning sensor device 200worn by a sport participant 1 may be acquired.

In this case, the image data and the positioning sensor data acquired bythe player tracking server 1000 may be data adjusted to be in timesynchronization with each other.

Also, the image data acquired by the player tracking server 1000 may bere-sampled data.

Also, the image data acquired by the player tracking server 100 may bedata from which noise is removed.

Also, the positioning sensor data acquired by the player tracking server1000 may be positioning sensor data that has undergone a processingprocess, such as noise removal, amplification, and filtering.

Also, the image data acquired by the player tracking server 1000 mayinclude location-related data of the sport participant and data relatedto a pixel corresponding to the sport participant.

Also, the player tracking server 1000 may additionally acquireidentification data related to the type of image capture device thatcaptures the image data and arrangement information related to thelocation, orientation, and the like of the image capture device.However, the above description is just an example, and the playertracking server 1000 may be implemented to include any suitable data.

Also, the positioning sensor data acquired by the player tracking server1000 may include location-related data of the sport participant,reliability-related information of a sensor signal, andidentifier-related data of the sport participant. However, the abovedescription is just an example, and the player tracking server 1000 maybe implemented to include any suitable data.

In the operation of determining the player's location (1300), theplayer's location may be determined based on at least one of the imagedata and the positioning sensor data.

As an example, the player tracking server 1000 may verify the validityof the image-based location computed from the image data and maydetermine that the player's location is the image-based location whenthe image-based location is valid. On the other hand, when theimage-based location computed from the image data is invalid, the playertracking server 1000 may determine that the player's location is theplayer's location computed from the positioning sensor data. This willbe described in detail with reference to FIGS. 17 to 19 .

As an example, the player tracking server 1000 may verify the validityof the positioning sensor-based location computed from the positioningsensor data and may determine that the player's location is thepositioning sensor-based location when the positioning sensor-basedlocation is valid. On the other hand, when the positioning sensor-basedlocation computed from the positioning sensor data is invalid, theplayer tracking server 1000 may determine that the player's location isthe image-based location computed from the image data. Alternatively,when the positioning sensor-based location computed from the positioningsensor data is invalid, the player tracking server 1000 may additionallyevaluate the validity of the image-based location and may determine theplayer's location on the basis of a result of evaluating the validity ofthe image-based location. This will be described in detail withreference to FIGS. 20 and 21 .

As another example, the player tracking server 1000 may determine theplayer's location by quantifying and comparing the reliability of thepositioning sensor-based location acquired from the positioning sensordata and the reliability of the image-based location acquired from theimage data. Specifically, the player tracking server 1000 may determinethe player's location on the basis of the one of the positioningsensor-based location and the image-based location that has higherreliability or may determine the player's location by assigning weightvalues corresponding to the reliability to the positioning sensor-basedlocation and the image-based location. This will be described in detailbelow with reference to FIG. 22 .

As another example, the player tracking server 1000 may determine theplayer's location on the basis of a reliability map of a positioningsensor-based location for each region of the playfield and a reliabilitymap of an image-based location for each region of the playfield. Morespecifically, the reliability of the computed positioning sensor-basedlocation of the player may differ for each region of the playfield.Also, the reliability of the computed image-based location of the playermay differ for each region of the playfield. Therefore, the playertracking server 1000 may select one of the positioning sensor-basedlocation and the image-based location on the basis of the reliabilitymap and determine the selected location as the player's location or maydetermine the player's location by assigning weight values correspondingto the reliability to the positioning sensor-based location and theimage-based location. This will be described in detail below withreference to FIGS. 23 to 25 .

When a location of a sport participant is determined from image dataacquired from an image capture device, it is possible to accuratelymeasure the location of the sport participant, and the influence ofstructures near a playfield is relatively small. Furthermore, it isrelatively easy to recognize or analyze the sport participant's motion,and it is possible to provide convenient use indoors. Also,conveniently, the sport participant does not need to wear a positioningsensor device. However, an occlusion event may occur between sportparticipants, and the amount of computation is relatively large becausethe size of image data is large.

On the other hand, when a location of a sport participant is determinedfrom positioning sensor data acquired from a positioning sensor device,an occlusion event that may occur in determining the location of thesport participant from image data does not occur, and the size of thepositioning sensor data is relatively smaller than that of the imagedata. Thus, the processing efficiency of the positioning sensor data maybe relatively higher than that of the image data.

Therefore, in the player tracking method according to an embodiment ofthe present disclosure, a location of a sport participant may bedetermined from image data. However, the location of the sportparticipant may be determined from the positioning sensor data only whenthe location of the sport participant computed from the image data isinvalid.

The following description will refer to FIG. 17 . FIG. 17 is a flowchartillustrating the player tracking method according to an embodiment ofthe present disclosure and is a diagram illustrating operation S1300 ofFIG. 16 in detail.

Referring to FIG. 17 , operation S1300 according to an embodiment of thepresent disclosure may include detecting an occlusion on the basis ofimage data (S1310), determining the severity of the occlusion (S1320),determining whether the severity of the occlusion is greater than orequal to a predetermined threshold value (S1330), determining a player'slocation on the basis of positioning sensor data or image data (S1340 orS1350), and outputting the determined location as the player's location(S1360).

In operation S1310, an occlusion event may be detected between sportparticipants included in the image data. Specifically, theabove-described player tracking server 1000 may detect an occlusionevent between sport participants included in the image data.Alternatively, the player tracking server 1000 may determine whether anocclusion event between sport participants included in the image datahas occurred.

In this case, when an occlusion event is not detected between the sportparticipants or it is determined that no occlusion event has occurredbetween the sport participants in operation S1310, the player trackingserver 1000 may determine the location of the sport participant on thebasis of a location computed from the image data.

On the other hand, when an occlusion event is detected between the sportparticipants or it is determined that an occlusion event has occurredbetween the sport participants in operation S1310, the operation ofdetermining the severity of the occlusion event (S1320) may beperformed. In operation S1320, the player tracking server 1000 mayadditionally determine the severity of the detected occlusion event.

As an example, the player tracking server 1000 may determine theseverity of the occlusion event in consideration of the degree to whichbounding boxes corresponding to the sport participants acquired from theimage data overlap as described above.

As another example, the player tracking server 1000 may determine theseverity of the occlusion event in consideration of a change in thetotal number of pixels corresponding to the sport participants acquiredfrom the image data as described above.

As another example, the player tracking server 1000 may determine theseverity of the occlusion event on the basis of the number of sportparticipants located within a predetermined region from a specific sportparticipant in consideration of the locations of the sport participantsacquired from the positioning sensor data as described above.

As another example, the player tracking server 1000 may determine theseverity of the occlusion event in consideration of team information ofthe sport participants related to the occlusion event as describedabove. The team information of the sport participants may be acquiredfrom pixel information corresponding to the uniforms of the sportparticipants in the image data and especially in an RGB map.Alternatively, the team information of the sport participants may beacquired from an identifier included in the positioning sensor data.

As another example, the player tracking server 1000 may determine theseverity of the occlusion event in consideration of a “situation” of agame in which the occlusion event is detected as described above.

In this case, the severity of the occlusion event may be quantified byany suitable method. Alternatively, an occlusion event may be classifiedby any suitable criterion according to the degree of severity.

In operation S1330, it may be determined whether a result of determiningthe severity of the occlusion event derived in operation S1320 isgreater than or equal to a predetermined value. Specifically, the playertracking server 1000 may determine whether the severity value of theocclusion event quantified in operation S1320 is greater than or equalto the predetermined value.

As an example, when the quantified severity value of the occlusion eventis greater than or equal to the predetermined value, it may bedetermined or predicted that the location of the sport participantcomputed based on the image data is invalid. Therefore, in operationS1350, when the quantified severity value of the occlusion event isgreater than or equal to the predetermined value, the player trackingserver 1000 may determine the location of the sport participant on thebasis of a positioning sensor-based location computed based on thepositioning sensor data rather than the image data.

As another example, when the quantified severity value of the occlusionevent is less than the predetermined value, it may be determined orpredicted that the location of the sport participant computed based onthe image data is valid. Therefore, in operation S1340, when thequantified severity value of the occlusion event is less than thepredetermined value, the player tracking server 1000 may determine thelocation of the sport participant on the basis of an image-basedlocation computed based on the image data.

Alternatively, the player tracking server 1000 may determine whether todetermine the player's location on the basis of the image data or on thebasis of the positioning sensor data according to the result ofclassifying the severity of the occlusion event in operation S1320.

As an example, the player tracking server 1000 may designate theocclusion event as a mild occlusion event or a severe occlusion eventdepending on the severity of the occlusion event. In this case, theplayer tracking server 1000 may determine the location of the sportparticipant on the basis of the image data according to the result ofdesignating the occlusion event as the mild occlusion event. On theother hand, the player tracking server 1000 may determine the locationof the sport participant on the basis of the positioning sensor dataaccording to the result of designating the occlusion event as the severeocclusion event.

In operation S1360, the location of the sport participant may be outputor updated on the basis of the determined location of the sportparticipant. The output location of the sport participant may be acoordinate value corresponding to any suitable coordinate system.

The following description will refer to FIG. 18 . FIG. 18 is a flowchartillustrating a training process for an artificial neural network relatedto a coordinate conversion algorithm according to an embodiment of thepresent disclosure.

A training method using an artificial neural network of a coordinateconversion unit according to an embodiment of the present disclosure mayinclude acquiring image data and an image-based location (S2100),acquiring sensor data and a positioning sensor-based location (S2200),verifying the image-based location (S2300), preparing a training set(S2400), and training the artificial neural network using the trainingset (S2500). The training method using the artificial neural network maybe implemented by the player tracking server 1000 according to anembodiment of the present disclosure or in any separate training module.

In operation S2100, image data transmitted from the image capture device300 may be acquired. Also, an image-based location may be acquired fromthe image data. The image-based location may be acquired from acoordinate value of a pixel corresponding to a location of a sportparticipant included in the image data.

In operation S2200, positioning sensor data transmitted from thepositioning sensor device 210 or 220 may be acquired. Also, apositioning sensor-based location may be acquired from the positioningsensor data. The positioning sensor-based location may be acquired fromlocation-related data included in the positioning sensor data or may beacquired from velocity- or acceleration-related data included in thepositioning sensor data. Specifically, in operation S2200, a positioningsensor-based location and a positioning sensor-based speed correspondingto the image-based location may be acquired.

In operation S2300, whether the image-based location acquired inoperation S2100 is usable as a training set may be verified based on theresult of determining the validity of the image-based location.

As an example, the player tracking server 1000 may determine thevalidity of the image-based location on the basis of whether theocclusion event is detected. In other words, in operation S2300, whetherthe image-based location is usable as a training set may be verifiedbased on the result of detecting the occlusion event in the image data.

For example, when an occlusion event is detected in the image data, itmay be determined that the acquired image-based location is notappropriate as a training set.

For example, when no occlusion event is detected in the image data (orwhen an occlusion event is detected and is determined not to be severe),it may be determined that the acquired image-based location is usable asa training set.

In operation S2400, the training set and the artificial neural networkmay be prepared according to the verification result of operation S2300.

The artificial neural network may include an input layer, an outputlayer, and a hidden layer including a plurality of nodes.

The training set may include the image-based location verified inoperation S2300 and a positioning sensor-based location and apositioning sensor-based speed corresponding to the image-basedlocation. In this case, the image-based location may be a locationrelated to a first coordinate system. Also, the positioning sensor-basedlocation may be a location related to a second coordinate systemdifferent from the first coordinate system.

In operation S2500, the artificial neural network may be trained usingthe training set prepared in operation S2400.

For example, the positioning sensor-based location and the positioningsensor-based speed of the training set may be input to the input layer.In this case, the input positioning sensor-based location andpositioning sensor-based speed may be converted from the secondcoordinate system to the first coordinate system through the artificialneural network and then may be output through the output layer. In thiscase, weight values of the nodes included in the hidden layer may beadjusted based on a difference between the image-based location of thetraining set and the output value output through the output layer.

By appropriately adjusting the weight values of the plurality of nodes,the artificial neural network may be trained such that errors betweenthe image-based location of the training set and output values outputthrough the output layer can be reduced.

FIG. 19 is a flowchart illustrating an example of the player trackingmethod through coordinate conversion using the trained artificial neuralnetwork of FIG. 18 .

The player tracking method through coordinate conversion using thetrained artificial neural network may include acquiring image data andan image-based location (S3100), acquiring positioning sensor data(S3200), determining the validity of the image-based location (S3300),and determining a location of a sport participant (S3400, S3500, orS3600).

In operation S3100, image data transmitted from the image capture device300 may be acquired. Also, an image-based location may be acquired fromthe image data. The image-based location may be acquired from acoordinate value of a pixel corresponding to a location of a sportparticipant included in the image data.

In operation S3200, positioning sensor data transmitted from thepositioning sensor device 200 may be acquired.

In operation S3300, the player tracking server 1000 may determine thevalidity of the image-based location.

As an example, according to any suitable method of determining thevalidity of the image-based location that has been described withreference to FIGS. 5 to 10 , the validity of the image-based locationacquired in operation S3100 may be determined or predicted.

For example, the validity of the image-based location may be determinedby the player tracking server 1000 in consideration of at least one ofthe presence of a detected occlusion event, a result of determining theseverity of the occlusion event, a vertical movement of a sportparticipant, and an internal disparity of the image-based location.

In operation S3400, when it is determined in operation S3300 that theimage-based location is valid, the player tracking server 1000 maydetermine the location of the sport participant on the basis of theimage-based location.

On the other hand, when it is determined in operation S3300 that theimage-based location is invalid, the player tracking server 1000 maydetermine the location of the sport participant on the basis of thepositioning sensor data.

In operation S3500, a positioning sensor-based location may be acquiredfrom the positioning sensor data acquired in operation S3200. Thepositioning sensor-based location may be acquired from location-relateddata included in the positioning sensor data or may be acquired fromvelocity- or acceleration-related data included in the positioningsensor data. Specifically, in operation S3500, a positioningsensor-based location and positioning sensor-based speeds correspondingto the image-based location may be acquired.

Referring to FIG. 19 , it is shown that the positioning sensor-basedlocation is acquired only when it is determined in operation S3300 thatthe image-based location is invalid, but this is just an example. It isobvious that the positioning sensor-based location or the positioningsensor-based speed may be computed from the positioning sensor data inoperation S3200 as well as in operation S3500.

In operation S3600, the positioning sensor-based location and thepositioning sensor-based speed acquired in operation S3500 may be inputas an input value of the input layer of the trained artificial neuralnetwork.

The positioning sensor-based location and the positioning sensor-basedspeed may be values acquired in the second coordinate system. Also, thepositioning sensor-based location and the positioning sensor-based speedmay be values of a location and a speed corresponding to the image-basedlocation determined to be invalid.

In operation S3600, as the positioning sensor-based location and thepositioning sensor-based speed of the second coordinate system input tothe input layer may pass through the artificial neural network,coordinates may be converted with respect to the image-based location ofthe first coordinate system.

The player tracking server 1000 may determine the location of the sportparticipant on the basis of the value output in operation S3600.

With the above-described player tracking method according to anembodiment according to the present disclosure, by tracking a player onthe basis of image data first, it is possible to facilitate recognitionof a behavior or situation of a sport participant, and it is alsopossible to implement location measurement with relatively highaccuracy, which are merits of the image data. However, in the case ofimage data, it may be difficult to distinguish objects due to occlusionevents, etc., and data processing efficiency may be slightly lowerbecause the data size is relatively large. However, such disadvantagesof image data can be compensated for with positioning sensor data.

When the location of the sport participant is determined frompositioning sensor data acquired from a positioning sensor device, it iseasy to distinguish between sport participants, and even at night, it ispossible to easily compute the locations of sport participants.Furthermore, the amount of computation is small because the data size isrelatively small. Also, conveniently, image capture devices do not haveto be installed in advance near a playfield or the like. However, sportparticipants need to wear positioning sensor devices, and the influenceof structures near a playfield may be great. Also, it may be relativelycomplicated to recognize events or behaviors of sport participantscompared to image data.

On the other hand, when a sport participant's movement is tracked basedon image data acquired from an image capture device, structures near aplayfield do not have any influence, and it is easy to recognize eventsor behaviors of sport participants.

Therefore, in another example of the player tracking method according toan embodiment of the present disclosure, a location of a sportparticipant may be determined from positioning sensor data. However, thelocation of the sport participant may be determined in furtherconsideration of the location of the sport participant computed fromimage data only when the location of the sport participant computed fromthe positioning sensor data is invalid.

The player tracking method according to another embodiment of thepresent disclosure may be implemented by the player tracking system 100shown in FIG. 3 . Also, the player tracking method according to anembodiment of the present disclosure may be implemented by the playertracking server 1000 shown in FIG. 4 .

The following description will refer to FIG. 20 . FIG. 20 is a flowchartillustrating a player tracking method according to an embodiment of thepresent disclosure. Referring to FIG. 20 , the player tracking methodaccording to an embodiment of the present disclosure may includeacquiring image data (S4100), acquiring positioning sensor data (S4200),and determining a player's location (S4300).

The details of operations S1100, S1200, and S1300 described withreference to FIG. 16 may be equally applied to operations S4100, S4200,and S4300. Therefore, the following description will focus on thedetails of FIG. 21 different from or added to those of FIGS. 16 to 19 .

The following description will refer to FIG. 21 . FIG. 21 is a flowchartillustrating the player tracking method according to an embodiment ofthe present disclosure and is a diagram illustrating operation S4300 ofFIG. 20 in detail.

Referring to FIG. 21 , operation S4300 according to an embodiment of thepresent disclosure may include determining whether the positioningsensor-based location is valid (S4310), verifying and predicting thevalidity of the image-based location (S4330 and S4340), determining theplayer's location on the basis of at least one of the positioningsensor-based location or the image-based location (S4320 or S4350), andoutputting the determined location as the player's location (S4360).

In operation S4310, whether the positioning sensor-based locationacquired from the positioning sensor data is valid may be determined.

As an example, the above-described player tracking server 1000 mayevaluate the validity of the positioning sensor-based location inconsideration of reliability-related data of the positioningsensor-based location included in the positioning sensor data.Specifically, reliability-related information (e.g., DoP, SNR) regardingthe positioning sensor-based location may be included in the positioningsensor data. In this case, the player tracking server 1000 may evaluatethe validity of the positioning sensor-based location in considerationof the reliability-related information acquired from the positioningsensor data and a preset reliability-related threshold value. Forexample, when the reliability-related information included in thepositioning sensor data is greater than the preset threshold value, theplayer tracking server 1000 may determine that the player's positioningsensor-based location computed from the positioning sensor data isvalid. On the other hand, when the reliability-related informationincluded in the positioning sensor data is smaller than the presetthreshold value, the player tracking server 1000 may determine that theplayer's positioning sensor-based location computed from the positioningsensor data is invalid.

As another example, the player tracking server 1000 may evaluate thevalidity of the positioning sensor-based location in consideration of aninternal disparity of the positioning sensor-based location acquiredfrom the positioning sensor data. Specifically, when a change between apositioning sensor-based location corresponding to a first time pointand a positioning sensor-based location corresponding to a second timepoint is greater than a predetermined threshold value, the playertracking server 1000 may determine that the positioning sensor-basedlocation computed from the positioning sensor data corresponding to thesecond time point is invalid.

As another example, the player tracking server 1000 may evaluate thevalidity of the positioning sensor-based location in consideration of adifference or disparity between the positioning sensor-based locationacquired from the positioning sensor data and the image-based locationacquired from the image data. This will be described in detail belowwith reference to FIG. 22 .

In operation S4320, when it is determined by the player tracking server1000 that the positioning sensor-based location is valid in operationS4310, the location of the sport participant may be determined based onthe positioning sensor-based location.

On the other hand, when it is determined by the player tracking server1000 that the positioning sensor-based location is invalid in operationS4310, the operations of verifying the validity of the image-basedlocation (S4330 and S4340) may be additionally included.

The operation of verifying the validity of the image-based location mayinclude verifying the validity of the image-based location (S4330) anddetermining whether the image-based location is predicted to be valid(S4340).

In operation S4330, the validity of the image-based location acquiredfrom the image data may be verified in consideration of at least one ofthe presence of an occlusion event having occurred in the image data,the severity of the occlusion event, the internal disparity of theimage-based location acquired from the image data, and a verticalmovement of the sport participant.

According to an embodiment, as described above, the player trackingserver 1000 may use various methods to detect an occlusion event ordetermine whether an occlusion event has occurred and may verify orpredict the validity of the image-based location on the basis of theocclusion event.

As an example, the player tracking server 1000 may detect an occlusionevent or determine whether an occlusion event has occurred inconsideration of the degree to which bounding boxes corresponding tosport participants acquired from the image data overlap.

As another example, the player tracking server 1000 may detect anocclusion event or determine whether an occlusion event has occurred inconsideration of data on pixels related to the sport participantacquired from the image data.

As another example, the player tracking server 1000 may detect anocclusion event or determine whether an occlusion event has occurred inconsideration of locations of the sport participants acquired from thepositioning sensor data. Specifically, the player tracking server 1000may determine that an occlusion event has occurred when a predeterminednumber or more of sport participants are located in a predeterminedregion from a positioning sensor-based location corresponding to aspecific sport participant.

As another example, the player tracking server 1000 may detect anocclusion event or determine whether an occlusion event has occurred inconsideration of locations of the sport participants acquired from thepositioning sensor data and data regarding a line-of-sight and anorientation of the image capture device.

In the player tracking method according to an embodiment of the presentdisclosure, when an occlusion event is detected or when it is determinedthat an occlusion has occurred, it may be predicted in operation S4340that the image-based location is invalid.

However, even though an occlusion event is detected or it is determinedthat an occlusion has occurred, when the severity of the occlusion isnot severe, it may be predicted in the operation S4340 that theimage-based location is valid. To this end, when an occlusion event hasoccurred, the validity of the image-based location may be verified orpredicted in additional consideration of the severity of the occlusionevent.

According to an embodiment, as described above, the player trackingserver 1000 may determine the severity of the occlusion event usingvarious methods and may verify or predict the validity of theimage-based location.

In the player tracking method according to an embodiment of the presentdisclosure, when a severe occlusion event is detected or when it isdetermined that a detected occlusion is severe, it may be predicted inoperation S4340 that the image-based location is invalid. On the otherhand, when a mild occlusion event is detected or when it is determinedthat a detected occlusion is mild, it may be predicted in operationS4340 that the image-based location is valid.

According to an embodiment, the player tracking server 1000 may verifyor predict the validity of the image-based location in consideration ofthe image-based location acquired from the image data and a verticalmovement of the sport participant at the same time point.

Referring to FIG. 9A again, when there is substantially no verticalmovement of a sport participant 1 included in image data, the differencebetween a location L1_image of the sport participant 1 computed from theimage data and an actual location L0 of the sport participant 1 may berelatively small. Therefore, the player tracking server 1000 may predictor determine that the location L1_image of the sport participant 1acquired from the image data is valid.

On the other hand, referring to FIG. 9B again, when there issubstantially a vertical movement of the sport participant 1 included inthe image data, the difference between a location L2_image of the sportparticipant 1 computed from the image data and an actual location L0 ofthe sport participant 1 may be relatively large. In other words, whenthere is substantially a vertical movement of the sport participant 1included in the image data, the location L2_image of the sportparticipant 1 computed from the image data is likely to have asignificant error with respect to the actual location L0. Therefore, theplayer tracking server 1000 may predictor determine that the locationL2_image of the sport participant 1 acquired from the image data isinvalid.

In the player tracking method according to an embodiment of the presentdisclosure, when a vertical movement of the sport participant isdetected, it may be predicted in operation S4340 that an image-basedlocation acquired at a time point when the vertical movement is detectedis invalid. On the other hand, when no vertical movement of the sportparticipant is detected, it may be predicted in S4340 that animage-based location is valid.

According to an embodiment, the player tracking server 1000 may verifyor predict the validity of the image-based location in consideration ofan internal disparity of the image-based location acquired from theimage data.

Referring to FIG. 10 again, when the difference between a locationLt1_image of the sport participant computed from a first time point anda location Lt2_image of the sport participant computed from a secondtime point is greater than a predetermined threshold value, the playertracking server 1000 may determine that the location Lt2_image of thesport participant computed from the second time point is invalid. On theother hand, when the difference between a location Lt3_image of thesport participant computed from a third time point and a locationLt4_image of the sport participant computed from a fourth time point issmaller than the predetermined threshold value, the player trackingserver 1000 may determine that the location Lt4_image of the sportparticipant computed from the fourth time point is valid.

In the player tracking method according to an embodiment of the presentdisclosure, when the internal disparity of the image-based location ismeasured to be greater than a predetermined threshold value, it may bepredicted in operation S4340 that the image-based location is invalid.On the other hand, when the internal disparity of the image-basedlocation is measured to be smaller than the predetermined thresholdvalue, it may be predicted in operation S4340 that the image-basedlocation is valid.

As described above, when it is predicted in operation S4340 that theimage-based location is valid, the player tracking server 1000 maydetermine the player's location on the basis of the image-based location(S4350).

On the other hand, when it is predicted in operation S4340 that theimage-based location is invalid, the player tracking server 1000 maydetermine the player's location on the basis of the positioningsensor-based location rather than the image-based location (S4320).

FIG. 21 shows that the player tracking server 1000 may determine theplayer's location on the basis of the positioning sensor-based locationrather than the image-based location (S4320) when it is predicted inoperation S4340 that the image-based location is invalid, but thepresent disclosure is not limited thereto. Accordingly, even when it ispredicted in operation S4340 that the image-based location is invalid,it is obvious that any suitable method such as the method of determiningthe player's location may be applied by the player tracking server 1000in consideration of both of the image-based location and the positioningsensor-based location (e.g., by assigning weight values corresponding toreliability).

In operation S4360, by outputting the determined location of the sportparticipant, the location of the sport participant may be output orupdated. The output location of the sport participant may be acoordinate value corresponding to any suitable coordinate system.

With the above-described player tracking method according to anembodiment according to the present disclosure, by tracking a player onthe basis of positioning sensor data first, it is possible to increasedata processing efficiency, and it is also possible to facilitate thedistinction between sport participants, which are merits of thepositioning sensor data. However, in the case of the positioning sensordata, the accuracy of location calculation may change depending on asurrounding structure, and it may be somewhat insufficient forrecognizing a behavior or situation of a sport participant.Advantageously, these disadvantages of the positioning sensor data canbe compensated for by the image data.

When the location of the sport participant is determined frompositioning sensor data acquired from a positioning sensor device, it iseasy to distinguish between sport participants, and even at night, it ispossible to easily compute the locations of sport participants.Furthermore, the amount of computation is small because the data size isrelatively small. Also, conveniently, image capture devices do not haveto be installed in advance near a playfield or the like.

When a location of a sport participant is determined from image dataacquired from an image capture device, it is possible to accuratelymeasure the location of the sport participant, and the influence ofstructures near a playfield is relatively small. Furthermore, it isrelatively easy to recognize or analyze the sport participant's motion,and it is possible to provide convenient use indoors. Also,conveniently, the sport participant does not need to wear a positioningsensor device.

Therefore, with the player tracking method according to an embodiment ofthe present disclosure, by determining a location of a sport participantin consideration of both of positioning sensor data and image data, itis possible to utilize advantages of both of the positioning sensor dataand the image data. Also, by determining a location of a sportparticipant by generating and comparing reliability indices of apositioning sensor-based location and an image-based location, it ispossible to more accurately track the sport participant.

The player tracking method according to an embodiment of the presentdisclosure may be implemented by the player tracking system 100 shown inFIG. 3 . Also, the player tracking method according to an embodiment ofthe present disclosure may be implemented by the player tracking server1000 shown in FIG. 4 .

The player tracking server 1000 according to an embodiment of thepresent disclosure may generate a reliability index related to theimage-based location and a reliability index related to the positioningsensor-based location. The player tracking server 1000 may be providedto generate the reliability index related to the image-based locationand the reliability index related to the positioning sensor-basedlocation.

Specifically, the player tracking server 1000 may generate a reliabilityindex for reliability related to a positioning sensor-based locationacquired from positioning sensor data. Specifically, the player trackingserver 1000 may generate the reliability index for the positioningsensor-based location on the basis of a result of evaluating thevalidity of the positioning sensor-based location.

As an example, information (e.g., DoP, SNR) related to the reliabilityof a sensor signal may be included in the positioning sensor data. Inthis case, the player tracking server 1000 may evaluate the validity ofthe positioning sensor-based location on the basis of the information(e.g., DoP, SNR) related to the reliability of the sensor signal, andthe player tracking server 1000 may generate the reliability index forthe positioning sensor-based location in consideration of theinformation (e.g., DoP, SNR) related to the reliability of the sensorsignal on the basis of the evaluation.

As another example, the player tracking server 1000 may evaluate thevalidity of the positioning sensor-based location on the basis of theinternal disparity of the positioning sensor-based location computedfrom the positioning sensor data. In this case, the player trackingserver 1000 may generate the reliability index for the positioningsensor-based location on the basis of a result of evaluating thevalidity of the positioning sensor-based location based on the internaldisparity of the positioning sensor-based location.

Also, the player tracking server 1000 may generate a reliability indexfor reliability related to an image-based location acquired from imagedata. Specifically, the player tracking server 1000 may generate thereliability index for the image-based location on the basis of a resultof evaluating the validity of the image-based location.

As an example, the player tracking server 1000 may evaluate the validityof the image-based location acquired from the image data according towhether an occlusion event is detected. In this case, the playertracking server 1000 may generate the reliability index for theimage-based location on the basis of a result of evaluating the validityof the image-based location according to whether an occlusion event isdetected.

As another example, the player tracking server 1000 may evaluate thevalidity of the image-based location acquired from the image data inconsideration of a vertical movement of the sport participant. In thiscase, the player tracking server 1000 may generate the reliability indexfor the image-based location on the basis of a result of evaluating thevalidity of the image-based location in consideration of the verticalmovement of the sport participant.

As another example, the player tracking server 1000 may evaluate thevalidity of the image-based location on the basis of the internaldisparity of the image-based location computed from the image data. Inthis case, the player tracking server 1000 may generate the reliabilityindex for the image-based location on the basis of a result ofevaluating the validity of the image-based location based on theinternal disparity of the image-based location.

In this case, the player tracking server 1000 may determine the locationof the sport participant by comparing the generated reliability indexfor the positioning sensor-based location and the generated reliabilityindex for the image-based location.

Here, in order to compare the reliability index for the positioningsensor-based location and the reliability index for the image-basedlocation, standardization is required between the reliability indices.Therefore, the player tracking server 1000 according to an embodiment ofthe present disclosure may perform the standardization between thereliability index related to the image-based location and thereliability index related to the positioning sensor-based location. Theplayer tracking server 1000 may perform the standardization with respectto the reliability index related to the image-based location and thereliability index related to the positioning sensor-based location.

The standardization of the reliability indices by the player trackingserver 1000 may be for correcting or standardizing the reliabilityindices using any suitable statistical method or standardization modelto compare the reliability index related to the image-based location andthe reliability index related to the positioning sensor-based location.

The following description will refer to FIG. 22 . FIG. 22 is a flowchartillustrating a player tracking method according to an embodiment of thepresent disclosure.

Referring to FIG. 22 , the player tracking method according to anembodiment of the present disclosure may include acquiring positioningsensor data and a positioning sensor-based location (S5100), acquiringimage data and an image-based location (S5200), determining an errorpossibility of at least one of the positioning sensor-based location andthe image-based location (S5300), and determining a player's location(S5400, S5500, and S5600). In this case, the operation of determining aplayer's location may include comparing reliability indices of thepositioning sensor-based location and the image-based location (S5500)and determining the player's location (S5600) and determining theplayer's location on the basis of one of the positioning sensor-basedlocation and the image-based location (S5400), depending on a result ofdetermining the error possibility in operation S5300.

In operation S5100, positioning sensor data transmitted from thepositioning sensor device 210 or 220 may be acquired. Also, apositioning sensor-based location may be acquired from the positioningsensor data. The positioning sensor-based location may be acquired fromlocation-related data included in the positioning sensor data or may beacquired from velocity- or acceleration-related data included in thepositioning sensor data.

In operation S5200, image data transmitted from the image capture device300 may be acquired. Also, an image-based location may be acquired fromthe image data. The image-based location may be acquired from acoordinate value of a pixel corresponding to a location of a sportparticipant included in the image data.

In operation S5300, it may be determined whether the positioningsensor-based location acquired in operation S5100 is valid and whetherthe image-based location acquired in operation S5200 is valid.Therefore, the embodiments of the player tracking server 1000 thatdetermines the validity of the image-based location and the positioningsensor-based location, which have been described with reference to FIGS.5 to 12 , may be implemented in operation S5400.

According to an embodiment, in operation S5300, an error possibility ofat least one of the positioning sensor-based location and the imagesensor-based location may be determined.

As an example, the error possibility of at least one of the positioningsensor-based location and the image-based location may be determined inconsideration of the difference between the positioning sensor-basedlocation and the image-based location.

For example, the player tracking server 1000 may compute a disparityindex between the image-based location of the sport participant acquiredfrom the image data and the positioning sensor-based location of thesport participant acquired from the positioning sensor data according toa time variable. In this case, the player tracking server 1000 maydetermine the validity of at least one of the image-based location andthe positioning sensor-based location on the basis of the disparityindex.

Specifically, the player tracking server 1000 may compute a firstdisparity index corresponding to the difference between a firstimage-based location and a first positioning sensor-based location whichare acquired at a first time point. Also, the player tracking server1000 may compute a second disparity index corresponding to thedifference between a second image-based location and a secondpositioning sensor-based location which are acquired at a second timepoint.

In this case, the player tracking server 1000 may evaluate the validityof at least one of the positioning sensor-based location (e.g., thefirst positioning sensor-based location and the second positioningsensor-based location) and the image-based location (e.g., the firstimage-based location and the second image-based location) inconsideration of the size of the first disparity index and the size ofthe second disparity index. For example, when the first disparity indexexceeds a predetermined first threshold value, the player trackingserver 1000 may determine that an error is present in at least one ofthe first image-based location and the first positioning sensor-basedlocation acquired at the first time point. On the other hand, when thesecond disparity index is less than a predetermined second thresholdvalue, the player tracking server 1000 may determine that one or both ofthe second image-based location and the second positioning sensor-basedlocation acquired at the second time point are valid. In this case, thefirst threshold value and the second threshold value may be the samevalue or may be predetermined to be different values.

Also, the player tracking server 1000 may evaluate the validity of atleast one of the positioning sensor-based location (e.g., the firstpositioning sensor-based location and the second positioningsensor-based location) and the image-based location (e.g., the firstimage-based location and the second image-based location) inconsideration of a change in the first disparity index and a change inthe second disparity index.

For example, when the variations of the first disparity index and thesecond disparity index exceed a predetermined third threshold value(preferably, when it is determined that the first image-based locationand the first positioning sensor-based location which are related to thefirst disparity index are valid), the player tracking server 1000 maydetermine that an error is present in at least one of the secondimage-based location and the second positioning sensor-based locationacquired at the second time point. On the other hand, when thevariations of the first disparity index and the second disparity indexare less than a predetermined fourth threshold value (preferably, whenit is determined that the first image-based location and the firstpositioning sensor-based location which are related to the firstdisparity index are valid), the player tracking server 1000 maydetermine that one or both of the second image-based location and thesecond positioning sensor-based location acquired at the second timepoint are valid. In this case, the third threshold value and the fourththreshold value may be the same value or may be predetermined to bedifferent values.

The above-described determination of the error possibility of at leastone of the image-based location and the positioning sensor-basedlocation in consideration of the difference between the image-basedlocation and the positioning sensor-based location is just an example,and the validity of the image-based location or the positioningsensor-based location may be determined using any suitable methodconsidering the difference between the image-based location and thepositioning sensor-based location.

When it is determined in operation S5300 that there is no errorpossibility of at least one of the positioning sensor-based location andthe image-based location, that is, when it is determined that both ofthe positioning sensor-based location and the image-based location arevalid, the player tracking server 1000 may determine the location of thesport participant on the basis of one of the positioning sensor-basedlocation and the image-based location in operation S5400.

Alternatively, since the positioning sensor-based location and theimage-based location have a small error possibility, the player trackingserver 1000 may determine the location of the sport participant inconsideration of both of the positioning sensor-based location and theimage-based location. For example, the player tracking server 1000 maydetermine the location of the sport participant using the average of thepositioning sensor-based location and the image-based location.Alternatively, the player tracking server 1000 may determine thelocation of the sport participant in consideration of the reliabilityindex of the positioning sensor-based location and the reliability indexof the image-based location.

When it is determined in operation S5300 that there is an errorpossibility of at least one of the positioning sensor-based location andthe image-based location, that is, when it is determined that at leastone of the positioning sensor-based location and the image-basedlocation is invalid, an operation of comparing the first reliabilityindex for the positioning sensor-based location and the secondreliability index of the image-based location may be performed by theplayer tracking server 1000 in operation S5500.

Specifically, the player tracking server 1000 may generate the firstreliability index for the positioning sensor-based location inconsideration of reliability-related information included in thepositioning sensor data. Alternatively, the player tracking server 1000may generate the first reliability index for the positioningsensor-based location on the basis of a result of determining thevalidity of the positioning sensor-based location based on the internaldisparity of the positioning sensor-based location.

Also, the player tracking server 1000 may generate the secondreliability index for the image-based location on the basis of a resultof determining the validity of the image-based location based on atleast one of an occlusion event related to the image-based location, avertical movement of the sport participant, and an internal disparity ofthe sport participant.

Also, the player tracking server 1000 may standardize the firstreliability index and the second reliability index.

In this case, in operations S5500 and S5600, the player tracking server1000 may determine the location of the sport participant by comparingthe first reliability index for the positioning sensor-based locationand the second reliability index for the image-based location.

As an example, the player tracking server 1000 may determine thelocation of the sport participant on the basis of a locationcorresponding to the higher one of the first reliability index and thesecond reliability index.

For example, when the first reliability index is larger than the secondreliability index, that is, when a positioning sensor-based locationcorresponding to the first reliability index has relativity highreliability, the player tracking server 1000 may determine the locationof the sport participant on the basis of the positioning sensor-basedlocation.

As another example, when the second reliability index is greater thanthe first reliability index, that is, when an image-based locationcorresponding to the second reliability index has relatively highreliability, the player tracking server 1000 may determine the locationof the sport participant on the basis of the image-based location.

As another example, the player tracking server 1000 may determine thelocation of the sport participant by assigning appropriate weight valuesto the positioning sensor-based location and the image-based location inconsideration of the first reliability index and the second reliabilityindex.

For example, when the first reliability index is greater than the secondreliability index, the player tracking server 1000 may determine thelocation of the sport participant by assigning a relatively high weightvalue to the positioning sensor-based location.

For example, when the second reliability index is greater than the firstreliability index, the player tracking server 1000 may determine thelocation of the sport participant by assigning a relatively high weightvalue to the image-based location.

For example, when the first reliability index is similar orsubstantially identical to the second reliability index, the playertracking server 1000 may determine the location of the sport participantby assigning similar weight values (e.g., a 1:1 weighted average) to thepositioning sensor-based location and the image-based location.

With the above-described player tracking method according to anembodiment of the present disclosure, by determining an errorpossibility of at least one of the positioning sensor-based location andthe image-based location, it is possible to accurately measure thelocation of the sport participant.

Also, even when at least one of the positioning sensor-based locationand the image-based location may have an error, the reliability of thepositioning sensor-based location and the image-based location may beevaluated. Thus, even in this case, it is possible to accurately measurethe location of the sport participant.

As described above, although the description was focused on determiningthe validity in consideration of the difference between the image-basedlocation and the positioning sensor-based location in operation S5300,this is just an example. The operation, which is related to FIGS. 5 to12 , of determining the validity of the image-based location and thepositioning sensor-based location may be equally applied.

In acquiring the location of the sport participant, the accuracy of thelocation of the sport participant may change depending on the region ofthe playfield where the sport participant is located. In other words,the reliability of the acquired location may change depending on theregion of the playfield where the sport participant is located. This isbecause reliability related to a sensor signal or image data may changefor each region of the playfield where the sport participant is located.

For example, the reliability of the location of the sport participantacquired from the positioning sensor data may be affected by thestructure or shape of a stadium near the playfield where the sportparticipant is located. Therefore, since the degree to which theinfluence of the structure or shape of the stadium may be different foreach region of the playfield, the reliability of the location of thesport participant acquired from the positioning sensor data may bedifferent for each region of the playfield.

Also, the reliability of the location of the sport participant acquiredfrom the image data may be affected by factors such as the location,angle, lens distortion, and the like of the image capture device.Therefore, the reliability of the location of the sport participantacquired from the image capture device may be different for each regionof the playfield.

Therefore, the play tracking method according to another embodiment ofthe present disclosure may generate a reliability map for each region ofthe playfield and use reliability information corresponding to aplayfield region where the sport participant is located. Therefore, withthe play tracking method according to an embodiment of the presentdisclosure, it is possible to more accurately acquire the location ofthe sport participant.

The player tracking method according to an embodiment of the presentdisclosure may be implemented by the player tracking system 100 shown inFIG. 3 . Also, the player tracking method according to an embodiment ofthe present disclosure may be implemented by the player tracking server1000 shown in FIG. 4 .

The player tracking server 1000 according to an embodiment of thepresent disclosure may generate a reliability index for each region ofthe playfield related to the image-based location and a reliabilityindex for each region of the playfield related to the positioningsensor-based location. Also, the player tracking server 1000 maygenerate reliability maps on the basis of the reliability index for eachregion of the playfield related to the image-based location and thereliability index for each region of the playfield related to thepositioning sensor-based location. Specifically, the player trackingserver 1000 may be provided to generate reliability maps on the basis ofthe reliability index for each region of the playfield related to theimage-based location and the reliability index for each region of theplayfield related to the positioning sensor-based location.

For each region of the playfield, the player tracking server 1000 maygenerate a reliability index related to the location of the sportparticipant acquired from the image data. In this case, the playertracking server 1000 may generate the reliability index for theimage-based location for each region of the playfield in considerationof the location of the sport participant on the playfield and a resultof evaluating the validity of the image-based location.

The details described above with reference to FIGS. 5 to 10 may besimilarly applied to the determination of the validity of theimage-based location.

For each region of the playfield, the player tracking server 1000 maygenerate a reliability index related to the location of the sportparticipant acquired from the positioning sensor data. In this case, theplayer tracking server 1000 may generate the reliability index for thepositioning sensor-based location for each region of the playfield inconsideration of the location of the sport participant on the playfieldand a result of evaluating the validity of the positioning sensor-basedlocation.

The details described above with reference to FIGS. 11 and 12 may besimilarly applied to the determination of the validity of thepositioning sensor-based location.

Also, the player tracking server 1000 may be provided to standardize thereliability index related to the image-based location and thereliability index related to the positioning sensor-based location. Thestandardization of the reliability indices by the player tracking server1000 may be for correcting or standardizing the reliability indexrelated to the image-based location and the reliability index related tothe positioning sensor-based location using any suitable statisticalmethod or standardization model to compare the reliability indices.

Also, the player tracking server 1000 may generate a first reliabilitymap on the basis of the generated and standardized reliability index foreach region of the playfield related to the positioning sensor-basedlocation. In this case, the first reliability map may includeinformation on a reliability index related to a positioning sensor-basedlocation acquired according to a region of the playfield.

The following description will refer to FIG. 23 . FIG. 23 is an examplediagram of the first reliability map according to an embodiment of thepresent disclosure. Referring to FIG. 23 , the first reliability map mayinclude reliability information related to the positioning sensor-basedlocation.

A plurality of regions including a first region R1_sensor, a secondregion R2_sensor, a third region R3_sensor, and a fourth regionR4_sensor included in the first reliability map may be regionscorresponding to a plurality of regions of the playfield.

Also, the reliability information related to the positioningsensor-based location may be different for each of the plurality ofregions included in the first reliability map.

As an example, reliability information related to the first regionR1_sensor of the first reliability map may have a relatively high value.For example, considering the structure of the stadium near theplayfield, a sensor signal with high reliability may be received when asport participant is located in the center of the playfieldcorresponding to the first region R1_sensor.

As another example, reliability information related to the second regionR2_sensor of the first reliability map may have a relatively low value.For example, considering the structure of the stadium near theplayfield, a sensor signal with relatively low reliability may bereceived when a sport participant is located at a corner of theplayfield corresponding to the second region R2_sensor because thesensor signal may be affected by the structure of the stadium.

As another example, reliability information related to the third regionR3_sensor or the fourth region R4_sensor of the first reliability mapmay have a value lower than reliability related to the first regionR1_sensor and higher than reliability related to the second regionR2_sensor.

However, the above-described reliability for each region of the firstreliability map is just an example for convenience of description, andit is obvious that the reliability for each region may vary depending onthe structure and shape of the stadium.

Also, the player tracking server 1000 may generate a second reliabilitymap on the basis of the generated and standardized reliability index foreach region of the playfield related to the image-based location.

In this case, the second reliability map may include information on areliability index related to an image-based location acquired accordingto a region of the playfield.

The following description will refer to FIG. 24 . FIG. 24 is an examplediagram of the second reliability map according to an embodiment of thepresent disclosure. Referring to FIG. 24 , the second reliability mapmay include reliability information related to the image-based location.

A plurality of regions including a first region R1_image, a secondregion R2_image, a third region R3_image, a fourth region R4_image, anda fifth region R5_image included in the second reliability map may beregions corresponding to a plurality of regions of the playfield.

Also, the reliability information related to the image-based locationmay be different for each of the plurality of regions included in thesecond reliability map.

As an example, reliability information related to the first regionR1_image of the second reliability map may have a relatively high value.For example, considering an orientation, which includes a location andan angle, of the image capture device 300, image data with relativelylow distortion may be acquired when a sport participant is located inthe region of the playfield corresponding to the first region R1_image.Therefore, when a sport participant is located in the region of theplayfield corresponding to the first region R1_image, the image-basedlocation acquired from the image data may have relatively highreliability.

As another example, reliability information related to the second regionR2_image of the second reliability map may have a relatively low value.For example, considering an orientation, which includes a location andan angle, of the image capture device 300, image data with relativelylarge distortion may be acquired when a sport participant is located inthe region of the playfield corresponding to the second region R2_image.Therefore, when a sport participant is located in the region of theplayfield corresponding to the second region R2_image, the image-basedlocation acquired from the image data may have relatively lowreliability.

As another example, considering an orientation, which includes alocation and an angle, of the image capture device 300, reliabilityinformation related to the third region R3_image, the fourth regionR4_image, or the fifth region R5_image of the second reliability map mayhave a value lower than reliability related to the first region R1_imageof the second reliability map and higher than reliability related to thesecond region R2_image.

However, the above-described reliability for each region of the secondreliability map is just an example for convenience of description, andit is obvious that the reliability for each region may vary depending onan arrangement, a number, lens distortion, and the like related to thelocation and orientation of the image capture device 300.

The following description will refer to FIG. 25 . FIG. 25 is a flowchartillustrating a player tracking method according to an embodiment of thepresent disclosure.

Referring to FIG. 25 , the player tracking method according to anembodiment of the present disclosure may include acquiring positioningsensor data and a positioning sensor-based location (S6100), acquiringreliability for each region of the positioning sensor-based location(S6200), acquiring image data and an image-based location (S6300),acquiring reliability for each region of the image-based location(S6400), generating a reliability map (S6500), and determining aplayer's location on the basis of the reliability of a reliability mapregion (S6600).

In operation S6100, positioning sensor data transmitted from thepositioning sensor device 210 or 220 may be acquired. Also, apositioning sensor-based location may be acquired from the positioningsensor data. The positioning sensor-based location may be acquired fromlocation-related data included in the positioning sensor data or may beacquired from velocity- or acceleration-related data included in thepositioning sensor data.

In operation S6200, reliability for each region of the positioningsensor-based location may be acquired.

In this case, the reliability for each region of the positioningsensor-based location may be generated by the player tracking server1000. For example, as described above, the player tracking server 1000may generate, for each region of the playfield, a reliability indexrelated to the location of the sport participant acquired from thepositioning sensor data. In this case, the player tracking server 1000may generate the reliability index for the positioning sensor-basedlocation for each region of the playfield in consideration of thelocation of the sport participant on the playfield and a result ofevaluating the validity of the positioning sensor-based location.

Also, the reliability for each region of the positioning sensor-basedlocation may be a reliability index standardized by the player trackingserver 1000.

The player tracking server 1000 may generate a reliability map relatedto the positioning sensor-based location on the basis of the acquiredreliability for each region of the positioning sensor-based location.

In operation S6300, image data transmitted from the image capture device300 may be acquired. Also, an image-based location may be acquired fromthe image data. The image-based location may be acquired from acoordinate value of a pixel corresponding to a location of a sportparticipant included in the image data.

In operation S6400, reliability for each region of the image-basedlocation may be acquired.

In this case, the reliability for each region of the image-basedlocation may be generated by the player tracking server 1000. Forexample, as described above, the player tracking server 1000 maygenerate, for each region of the playfield, a reliability index relatedto the location of the sport participant acquired from the image data.In this case, the player tracking server 1000 may generate thereliability index for the image-based location for each region of theplayfield in consideration of the location of the sport participant onthe playfield and a result of evaluating the validity of the image-basedlocation.

Also, the reliability for each region of the image-based location may bea standardized reliability index.

The player tracking server 1000 may generate a reliability map relatedto the image-based location on the basis of the acquired reliability foreach region of the image-based location.

In operation S6500, the player tracking server 1000 may generate thefirst reliability map related to the positioning sensor-based locationand the second reliability map related to the image-based location inconsideration of the reliability for each region of the positioningsensor-based location and the reliability for each region of theimage-based location.

It has been described that the player tracking server 1000 generates thefirst reliability map related to the positioning sensor-based locationand the second reliability map related to the image-based location, butthis is just an example. The player tracking server 1000 may generate asingle reliability map in consideration of both of the reliability foreach region of the positioning sensor-based location and the reliabilityfor each region of the image-based location.

In operation S6600, the player tracking server 1000 may determine thelocation of the sport participant in consideration of the reliabilityinformation included in the first reliability map related to thepositioning sensor-based location and the second reliability map relatedto the image-based location, which are generated in operation S6500.

To this end, locations of the sport participant corresponding to theregion of the first reliability map and the region of the secondreliability map should be additionally considered.

Therefore, additionally, the player tracking server 1000 may beimplemented to further acquire information regarding a region of theplayfield where the sport participant is located.

For example, the player tracking server 1000 may acquire data related tothe location of the sport participant from the positioning sensor data,and the player tracking server 1000 may detect the region of theplayfield where the sport participant is located on the basis of datarelated to the location of the sport participant included in thepositioning sensor data.

As another example, the player tracking server 1000 may acquire datarelated to the location of the sport participant from the image data,and the player tracking server 1000 may detect the region of theplayfield where the sport participant is located on the basis of datarelated to the location of the sport participant included in the imagedata.

The player tracking server 1000 may determine the location of the sportparticipant by sequentially considering the first reliabilityinformation of the first reliability map region and the secondreliability information of the second reliability map regioncorresponding to the detected region where the sport participant islocated.

As an example, the player tracking server 1000 may determine thelocation of the sport participant in consideration of a locationacquired from data having the higher one of the first reliabilityinformation and the second reliability information.

For example, when a sport participant is located in a region where thefirst reliability information is higher than the second reliabilityinformation, the player tracking server 1000 may determine the locationof the sport participant on the basis of the positioning sensor-basedlocation.

As another example, when a sport participant is located in a regionwhere the second reliability information is higher than the firstreliability information, the player tracking server 1000 may determinethe location of the sport participant on the basis of the image-basedlocation.

As another example, the player tracking server 1000 may determine thelocation of the sport participant by assigning weight values accordingto the first reliability information and the second reliabilityinformation.

For example, when a sport participant is located in a region where thefirst reliability information is higher than the second reliabilityinformation, the player tracking server 1000 may determine the locationof the sport participant by assigning a higher weight value to thepositioning sensor-based location than to the image-based location.

As another example, when a sport participant is located in a regionwhere the second reliability information is higher than the firstreliability information, the player tracking server 1000 may determinethe location of the sport participant by assigning a higher weight valueto the image-based location than to the positioning sensor-basedlocation.

The above-described reliability map may be generated or updated during asport game in real time. However, this is just an example, andreliability maps related to the image-based location and the positioningsensor-related location may be pre-generated even before a sport game.

In the above, an embodiment in which an image-based location is acquiredfrom at least one image capture device and the accuracy of the locationof the sport participant is increased by additionally considering thepositioning sensor-based location has been described.

A player tracking method, a player tracking device, and a playertracking system that increase the accuracy of a location of a sportparticipant on the basis of a plurality of pieces of image data acquiredfrom a plurality of image capture devices will be described in detailbelow. Although the description was focused on the player trackingsystem 100 according to this embodiment increasing the accuracy of thelocation of the sport participant on the basis of a plurality of piecesof image data acquired from a plurality of image capture devices, it isobvious that the positioning sensor-based location can be additionallyconsidered in combination.

FIG. 26 is a schematic diagram illustrating the player tracking device,the player tracking system, and the player tracking method according toan embodiment of the present disclosure.

Referring to FIG. 26 , with the player tracking method, the playertracking device, and the player tracking system according to anembodiment of the present disclosure, a player's location may be trackedusing image data acquired from a plurality of image capture devices.

A player tracking system 100 according to an embodiment of the presentdisclosure will be described below with reference to FIG. 26 .

Referring to FIG. 26 , the player tracking system 100 may include aplurality of image capture devices 301, 302, 303, and 304, a positioningsensor device 200, and a player tracking server 2000. Meanwhile, FIG. 26does not show the intermediate server 400 or the play field shown inFIG. 3 , but this is for convenience of description, image data orpositioning sensor data may be acquired through the intermediate server400 of the player tracking system 100 of FIG. 3 .

The plurality of image capture devices 301, 302, 303, and 304 of theplayer tracking system 100 may capture a playfield and a sportparticipant 1, and the player tracking device 2000 may continuouslytrack the location of the sport participant 1 by determining thelocation of the sport participant 1 on the basis of image data acquiredfrom the plurality of image capture devices 301, 302, 303, and 304.

Referring to FIG. 26 , the plurality of image capture devices 301, 302,303, and 304 may be located around the playfield. The plurality of imagecapture devices 301, 302, 303, and 305 may capture image data includingthe playfield and a plurality of sport participants 1 and transmit theimage data to the player tracking device 2000 according to an embodimentof the present disclosure.

The plurality of image capture devices 301, 302, 303, and 304 may bearranged at different locations around the playfield as shown in FIG. 26. However, this is just an example, and the plurality of image capturedevices 301, 302, 303, and 304 may be arranged at substantially the sameplace or adjacent locations near the playfield. Also, it is obvious thatthe plurality of image capture devices 301, 302, 303, and 304 may bearranged at any suitable location in various ways.

In this case, the player tracking device 2000 may additionally acquire,from a plurality of image capture devices, data related to lensdistortion parameters of the plurality of image capture devices andarrangement information related to the locations and orientations of theplurality of image capture devices. Thus, image data that allows atarget sport participant to be detected well or that allows a locationof a target sport participant to be accurately acquired may be selected.This will be described in detail with reference to FIGS. 27 and 28 .

In this case, the image data acquired from the plurality of imagecapture devices may be labeled with information regarding the imagecapture devices and transmitted to the player tracking device 2000. Forexample, any suitable labeling technique may be applied to include anysuitable data indicating that the first image data acquired from thefirst image capture device 301 is captured from the first image capturedevice 301. Also, any suitable labeling technique may be applied toinclude any suitable data indicating that the second image data acquiredfrom the second image capture device 302 is captured from the secondimage capture device 302. Thus, an image capture device capable ofacquiring image data that allows a target sport participant to bedetected well or that allows a location of a target sport participant tobe accurately acquired may be determined to track the sport participantthrough an occlusion prediction. This will be described in detail withreference to FIGS. 29 and 30 .

In this case, a plurality of sport participants 1 may wear a positioningsensor device 200, and the player tracking server 2000 may acquirepositioning sensor data as described above with reference to FIG. 3 .The positioning sensor data related to the plurality of sportparticipants 1 may be considered in detecting the presence of anocclusion, in generating a virtual playfield, or in detecting anocclusion. This will be described in detail with reference to FIGS. 27to 31 .

The player tracking device 2000 may be implemented as a server device ofany suitable type. Hereinafter, the player tracking device 2000 isreferred to as the player tracking server 2000. However, this is justfor convenience of description, and the player tracking device 2000 isnot necessarily implemented in the form of a server.

The player tracking server 2000 according to an embodiment of thepresent disclosure will be described below.

According to an embodiment of the present disclosure, the playertracking server 2000 may acquire a plurality of pieces of image datafrom a plurality of image capture devices 301, 302, 303, and 304 inorder to determine the location of the sport participant. Also, theplayer tracking server 2000 may select an image including a target sportparticipant to be tracked from among the plurality of pieces of imagedata as a candidate image. Also, the player tracking server 2000 mayselect a valid image from a plurality of acquired candidate images inconsideration of an occlusion event. Also, the player tracking server2000 may determine the location of the target sport participant from theselected valid image.

The details described with reference to FIG. 4 may be equally applied tothe player tracking server 2000 according to an embodiment of thepresent disclosure. That is, the player tracking server 2000 may includea communication module 1100, a memory 1200, and a controller 1300. Inparticular, the details described with reference to FIG. 4 may beequally applied to the communication module 1100, the memory 1200, andthe controller 1300, and thus a detailed description thereof will beomitted. Therefore, the following description will focus on detailsadded to the player tracking server 2000.

Some operations performed by an example of the player tracking system100 in association with the player tracking method will be described indetail below.

The player tracking system 100 according to this embodiment may acquirea plurality of pieces of image data from a plurality of image capturedevices. Specifically, the player tracking server 2000 may acquire aplurality of pieces of image data captured by a plurality of imagecapture devices.

The player tracking system 100 according to an embodiment of the presentdisclosure may select a candidate image from the plurality of pieces ofimage data. Specifically, the player tracking server 2000 may beimplemented to detect a target sport participant to be tracked in eachof the plurality of pieces of image data acquired from the plurality ofimage capture devices 301, 302, 303, and 304. Specifically, theplurality of pieces of image data may include data (e.g., pixels)corresponding to a plurality of sport participants. In this case, theplayer tracking server 2000 may be implemented to recognize the sportparticipants in consideration of the data corresponding to the pluralityof sport participants and may detect a target sport participant inadditional consideration of data related to identifiers corresponding tothe recognized sport participants.

Also, the player tracking server 2000 may be implemented to select atleast one candidate image from the plurality of pieces of image dataaccording to a result of detecting the target sport participant. Forexample, the player tracking server 2000 may be implemented to selectimage data in which the target sport participant is detected from amongthe plurality of pieces of image data as a candidate image. Thecandidate image may include at least one candidate image.

The player tracking system 100 according to an embodiment of the presentdisclosure may be implemented to select a valid image that may beconsidered in determining the location of the target sport participant.Specifically, the player tracking server 2000 may be implemented toselect a valid image that may be considered in determining the locationof the target sport participant from among the at least one selectedcandidate image.

Here, the valid image may be used in the sense of encompassing an imagethat may be considered in determining the location of the target sportparticipant. For example, an image in which the target sport participantis detected may be a valid image. However, according to a preferredembodiment, in order to more accurately determine the location of thetarget sport participant, the valid image may be defined as an image inwhich no occlusion event is detected in relation to the target sportparticipant or an image in which an occlusion event is detected but theseverity of the occlusion event is not severe.

As an example, the player tracking server 2000 may select at least onevalid image from among the at least one candidate image in considerationof whether an occlusion event has occurred in the plurality of pieces ofimage data or a result of determining whether an occlusion event hasoccurred.

For example, the player tracking server 2000 may be implemented toselect, as valid images, candidate images in which an occlusion eventrelated to a target sport participant is not detected.

As another example, the player tracking server 2000 may be implementedto select, as valid images, mild candidate images in which an occlusionevent related to a target sport participant is detected but the severityof the occlusion event is not severe.

As another example, the player tracking server 2000 may be implementedto select valid images from among candidate images in which an occlusionevent related to a target sport participant is detected in considerationof whether the target sport participant hides or is hidden by anothersport participant.

Specifically, even though an occlusion event related to a target sportparticipant is detected in a first candidate image, the determination ofthe location of the target sport participant from the first candidateimage may have relatively high accuracy when the occlusion event isdetected by means of the target sport participant hiding another sportparticipant.

On the other hand, even though an occlusion event related to a targetsport participant is detected in a second candidate image, thedetermination of the location of the target sport participant from thesecond candidate image may be relatively difficult when the occlusionevent is detected by means of the target sport participant being hiddenby another sport participant. Also, the location of the target sportparticipant acquired from the second candidate image may have relativelylow accuracy.

Therefore, the player tracking server 2000 may be implemented to selecta valid image in consideration of whether the occlusion event isdetected in the candidate image by means of the target sport participanthiding or being hidden by another sport participant. Preferably, theoperation of the player tracking server 2000 may be advantageous whenthe occlusion event related to the target sport participant is detectedin most or all of the candidate images.

As another example, the player tracking server 2000 may be implementedto select valid images in consideration of arrangement informationrelated to the location and orientations of the plurality of imagecapture devices 301, 302, 303, and 304. Specifically, the plurality ofimage capture devices 301, 302, 303, and 304 may be placed in anysuitable arrangement at different locations around the playfield. Inthis case, the player tracking server 2000 may be implemented to select,as a valid image, image data in which the target sport participant islargest or image data in which the target sport participant is locatedclosest to the center from among the plurality of pieces of image dataacquired from the plurality of image capture devices 301, 302, 303, and304 in consideration of the arrangement information of the plurality ofimage capture devices 301, 302, 303, and 304.

As another example, the player tracking server 2000 may be implementedto select a valid image in consideration of a lens distortion parameter.Specifically, there is a high possibility that a relatively small amountof distortion occurs at the center of the image data when the lensdistortion parameter is considered. Accordingly, when the location ofthe target sport participant is computed from image data where thetarget sport participant is located at the center of the image data, theaccuracy of the location may be relatively high. On the other hand,there is a high possibility that a relatively large amount of distortionoccurs at the edge of the image data when the lens distortion parameteris considered. Accordingly, when the location of the target sportparticipant is computed from image data where the target sportparticipant is located at the edge of the image data, the accuracy ofthe location may be relatively low. Accordingly, the player trackingserver 2000 may be implemented to select, as a valid image, image datain which a region where the target participant is located has arelatively small amount of distortion in consideration of the lensdistortion parameter.

The following description will refer to FIG. 27 . FIG. 27 is a diagramillustrating an exemplary process of the player tracking server 2000selecting a valid image according to an embodiment of the presentdisclosure.

FIG. 27A shows a plurality of pieces of image data acquired from theplurality of image capture devices 301, 302, 303, and 304. In this case,the plurality of pieces of image data of FIG. 27A may be pieces of imagedata acquired at the same time point. The plurality of pieces of imagedata of FIG. 27A may include image data including a target sportparticipant whose location is to be determined and image data notincluding the target sport participant.

FIG. 27B shows at least one candidate image in which the target sportparticipant is detected. Specifically, the player tracking server 2000may detect a target sport participant in each of the plurality of piecesof image data of FIG. 27A and may select image data in which the targetsport participant is detected as a candidate image.

FIG. 27C shows at least one or more valid images. Specifically, thevalid images may be images in which an occlusion event related to thetarget sport participant is not detected among the candidate images ofFIG. 27B.

When the valid image includes a plurality of valid images as shown inFIG. 27C, the player tracking server 2000 may determine the location ofthe target sport participant in consideration of the location of thetarget sport participant acquired from each of the valid images. Forexample, the player tracking server 2000 may determine the location ofthe target sport participant by averaging the locations of the targetsport participant acquired from the valid images.

As another example, when the valid image includes a plurality of validimages as shown in FIG. 27C, the player tracking server 2000 maydetermine the location of the target sport participant in furtherconsideration of arrangement information, lens distortion parameters, orthe like of the image capture devices.

FIG. 27D shows an example of at least a single image selected by theplayer tracking server 2000 in consideration of arrangement information,lens distortion parameters, or the like of the image capture devices.

The player tracking system 100 according to an embodiment of the presentdisclosure may determine the location of the target sport participant onthe basis of a valid image. Specifically, the player tracking server2000 according to an embodiment of the present disclosure may determinea location on the basis of pixel data (a pixel location) correspondingto a target sport participant of a selected valid image.

In an embodiment, when the selected valid image includes a plurality ofvalid images, the player tracking server 2000 may determine the locationof the target sport participant in consideration of the locations of thetarget sport participant acquired from the plurality of valid images.

Specifically, the valid image may include a first valid image and asecond valid image. In this case, the player tracking server 2000 mayacquire a first location of the target sport participant on the basis ofa pixel location corresponding to the target sport participant in thefirst valid image. Also, the player tracking server 2000 may acquire asecond location of the target sport participant on the basis of a pixellocation corresponding to the target sport participant in the secondvalid image.

Finally, the player tracking server 2000 may determine the location ofthe target sport participant on the basis of both of the first locationand the second location.

As an example, the player tracking server 2000 may determine thelocation of the target sport participant by averaging the first locationand the second location.

In another embodiment, when the selected valid image includes aplurality of valid images, the player tracking server 2000 may determinethe location of the target sport participant in consideration of thelocation of the target sport participant and the location of the imagecapture device that captures the valid image. To this end, the playertracking server 2000 may be provided to select a single image from validimages in consideration of conditions related to the location of thetarget sport participant and the location of the image capture devicethat captures the valid image. In this case, the player tracking server2000 may determine the location of the target sport participant on thebasis of the single image.

For example, it is likely that the location of the target sportparticipant acquired from the valid image is relatively accurate whenthe location of the target sport participant and the location of theimage capture device that captures the valid image are relatively closeto each other. On the other hand, it is unlikely that the location ofthe target sport participant acquired from the valid image is relativelyaccurate when the location of the target sport participant and thelocation of the image capture device that captures the valid image arerelatively far from each other.

Therefore, when the selected valid image includes a plurality of validimages, the player tracking server 2000 may determine the location ofthe target sport participant in additional consideration of theconditions related to the location of the target sport participant andthe location of the image capture device that captures the valid image.

In another embodiment, when the selected valid image includes aplurality of valid images, the player tracking server 2000 may determinethe location of the target sport participant in consideration of a lensdistortion parameter of the image capture device that captures the validimage. To this end, the player tracking server 2000 may be provided toselect a single image from valid images in consideration of conditionsrelated to the lens distortion parameter of the image capture devicethat captures the valid image. In this case, the player tracking server2000 may determine the location of the target sport participant on thebasis of the single image.

For example, with regard to the lens distortion parameter of the imagecapture device that captures the valid image, when the target sportparticipant is located in a region of the valid image where lensdistortion is relatively large (e.g., in an edge portion of the validimage), the location acquired from the target sport participant may havelow accuracy.

On the other hand, with regard to the lens distortion parameter of theimage capture device that captures the valid image, when the targetsport participant is located in a region of the valid image where lensdistortion is relatively small (e.g., in a center portion of the validimage), the location acquired from the target sport participant may havehigh accuracy.

Therefore, when the selected valid image includes a plurality of validimages, the player tracking server 2000 may determine the location ofthe target sport participant in additional consideration of conditionsrelated to the lens distortion parameter of the image capture devicethat captures the valid image.

The player tracking server 2000 according to an embodiment of thepresent disclosure may determine the location of the target sportparticipant in consideration of whether the target sport participanthides another sport participant or is hidden by another sportparticipant. Preferably, the determination of the location of the targetsport participant in consideration of whether the target sportparticipant hides or is hidden by another sport participant may beadvantageously applied to a case in which there is no valid image, thatis, a case in which the target sport participant is detected from aplurality of pieces of image data and an occlusion event related to thetarget sport participant has occurred in all candidate images. To thisend, the player tracking server 2000 may be provided to select a targetimage from the candidate images in consideration of whether the targetsport participant hides or is hidden by another sport participant. Inthis case, the player tracking server 2000 may determine the location ofthe target sport participant on the basis of the target image.

For example, when the target sport participant hides another sportparticipant, a pixel location corresponding to the target sportparticipant in the image data may be specified. Therefore, the playertracking server 2000 may select image data in which the target sportparticipant hides another sport participant as a target image. Also, theplayer tracking server 2000 may determine the location of the targetsport participant on the basis of the selected target image.

On the other hand, when the target sport participant is hidden byanother sport participant, it is highly likely that a pixel locationcorresponding to the target sport participant in the image data is notspecified. Therefore, the player tracking server 2000 may be implementedsuch that the image data in which the target sport participant is hiddenby another sport participant is not used to determine the location ofthe target sport participant.

However, the above description is just an example, and it is obviousthat when an occlusion event related to the target sport participantoccurs in all candidate images, the player's location may be determinedusing the positioning sensor-based location which has been describedwith reference to FIGS. 16 to 25 .

The configuration and operation of the player tracking server 2000according to an embodiment of the present disclosure have been describedabove. A player tracking method according to this embodiment will bedescribed below. In the following description, the player trackingmethod according to this embodiment is performed by the above-describedplayer tracking system 100. However, this is just for convenience ofdescription, and thus the player tracking method according to thisembodiment is not limited to the above-described player tracking system100. That is, the player tracking method, which will be described below,does not necessarily have to be performed only by the player trackingsystem 100 but may be performed by another system or device having afunction similar to that of the above-described player tracking system100.

The following description will refer to FIG. 28 . FIG. 28 is a flowchartof a player tracking method according to an embodiment of the presentdisclosure.

Referring to FIG. 28 , the player tracking method according to anembodiment of the present disclosure may include acquiring a pluralityof pieces of image data (S7100), selecting a piece of image dataincluding a target sport participant (S7200), selecting a valid image(S7300), and determining a location of the target sport participant onthe basis of the valid image (S7400).

The above-described operations will be described in detail below.

The player tracking server 2000 may acquire a plurality of pieces ofimage data (S7100). Specifically, the player tracking server 2000 mayacquire a plurality of pieces of image data from the plurality of imagecapture devices 301, 302, 303, and 304 through the communication module1100.

In operation S7100, the player tracking server 2000 may acquire aplurality of pieces of image data captured by the plurality of imagecapture devices 301, 302, 303, and 304. In this case, the playertracking server 2000 may also acquire arrangement information related tolocations and orientations of the plurality of image capture devices301, 302, 303, and 304, identifiers of the image capture devices, lensdistortion parameters, and location-related data (positioning sensordata or pixel data of image data) of a sport participant.

The plurality of pieces of image data acquired in operation S7100 mayinclude a plurality of sport participants. The plurality of pieces ofimage data may or may not include a target sport participant whoselocation is to be determined.

Therefore, the player tracking server 2000 may detect the target sportparticipant from the plurality of pieces of image data and select imagedata including the target sport participant (S7200). Specifically, theplayer tracking server 2000 may be provided to perform an operation ofdetecting the target sport participant included in the plurality ofpieces of image data. In this case, the player tracking server 2000 maybe implemented to select a candidate image according to a result ofdetecting the target sport participant.

For example, the player tracking server 2000 may select, as a candidateimage, image data in which the target sport participant is detected.

In this case, the candidate image in which the target sport participantis detected may include at least one candidate image.

As an example, the target sport participant may be detected in all ofthe plurality of pieces of image data. In this case, all images in whichthe target sport participant is detected may be selected as candidateimages.

However, the candidate images selected according to whether the targetsport participant is detected may include images that are not suitableas a basis for determining the location of the target sport participant.

Therefore, the player tracking server 2000 may select a valid image fromthe candidate images (S7300). Here, the valid image may be used in thesense of encompassing any image suitable for determining the location ofthe target sport participant.

Specifically, the player tracking server 2000 may be provided to selecta valid image from among the candidate images in consideration ofwhether an occlusion event related to the target sport participant isdetected.

As an example, the player tracking server 2000 may detect the occlusionevent or determine whether the occlusion event has occurred.Alternatively, the player tracking server 2000 may determine theseverity of the occlusion event.

In particular, the player tracking server 2000 may detect an occlusionevent related to the target sport participant or determine whether theocclusion event has occurred. Alternatively, the player tracking server2000 may determine the severity of the occlusion event related to thetarget sport participant.

In this case, the player tracking server 2000 may select at least onevalid image from the candidate images selected in operation S7200 on thebasis of a result of detecting the occlusion event related to the targetsport participant.

For example, the player tracking server 2000 may select, as validimages, candidate images in which the occlusion event related to thetarget sport participant is not detected.

On the other hand, the player tracking server 2000 may not select, asvalid images, candidate images in which the occlusion event related tothe target sport participant is detected. However, even though theocclusion event related to the target sport participant is detected, theplayer tracking server 2000 may be provided to select a correspondingcandidate image as a valid image when it is determined that theocclusion event is not severe.

In operation S7300, in consideration of additional conditions, a singleimage may be selected from valid images selected in consideration of aresult of detecting the occlusion event.

Specifically, when a plurality of valid images are selected inconsideration of the result of detecting the occlusion event related tothe target sport participant, a single image may be selected from thevalid images in additional consideration of a condition related to thelocation of the target sport participant and the location of the imagecapture device that captures the valid image or conditions related to alens distortion parameter of the image capture device that captures thevalid image.

As an example, the accuracy of the location of the target sportparticipant may be different depending on the size or sharpness of thetarget sport participant included in the valid image. When the targetsport participant included in the valid image has a large size or a highresolution, the location of the target sport participant determined fromthe corresponding valid image may be relatively accurate.

In this case, the size or resolution of the target sport participantincluded in the valid image may depend on the location of the targetsport participant and a location of the image capture device thatcaptures the valid image. For example, when the location of the targetsport participant and the location of the image capture device thatcaptures the valid image are relatively close to each other, thecaptured target sport participant may be large and clear. On the otherhand, when the location of the target sport participant and the locationof the image capture device that captures the valid image are relativelyfar from each other, the captured target sport participant may berelatively small and unclear.

Therefore, according to an embodiment, in operation S7300, the playertracking server 2000 may be implemented to select a single image fromamong the valid images in consideration of a condition related to thelocation of the target sport participant and the location of the imagecapture device that captures the valid image.

As another example, the accuracy of the location of the target sportparticipant may be different depending on the location of the targetsport participant in the valid image and the degree of distortion foreach region of the valid image.

For example, when the target sport participant is located in a regionwith less distortion of the valid image, the location of the targetsport participant determined from the corresponding valid image may berelatively accurate.

As another example, when the target sport participant is located in thecenter of the valid image where the degree of distortion is relativelylow, the accuracy of the location of the target sport participantdetermined from the corresponding valid image may be relatively high. Onthe other hand, when the target sport participant is located at an edgeof the valid image where the degree of distortion is relatively high,the accuracy of the location of the target sport participant determinedfrom the corresponding valid image may be relatively low.

In this case, the degree of distortion for each region of the validimage may depend on a lens distortion parameter. Therefore, according toan embodiment, in operation S7300, the player tracking server 2000 maybe implemented to select a single image from among the valid images inconsideration of a condition related to a lens distortion parameter ofthe image capture device that captures the valid image.

In operation S7300, a target image may be selected from the candidateimages selected in operation S7200 in consideration of the type of theocclusion event related to the target sport participant. Preferably, theselection of the target image in consideration of the type of theocclusion event related to the target sport participant may beadvantageously applied to a case in which the occlusion event related tothe target sport participant is detected in all of the candidate images.

The type of occlusion event may include a first type in which the targetsport participant hides another sport participant and a second type inwhich the target sport participant is hidden by another sportparticipant.

In the case of the first type, it may be relatively easy to specify alocation of a pixel corresponding to the target sport participant. Onthe other hand, in the case of the second type, it may be relativelydifficult to specify a location of a pixel corresponding to the targetsport participant because the target sport participant is hidden byanother sport participant.

Therefore, the player tracking server 2000 may be provided to select acandidate image corresponding to the first type as a target image. Onthe other hand, the player tracking server 2000 may be provided not toselect a candidate image corresponding to the second type as a targetimage.

The player tracking server 2000 may determine the location of the targetsport participant on the basis of the selected target image (S7400).

Specifically, the player tracking server 2000 may be provided todetermine the location of the target sport participant in considerationof a pixel location corresponding to the target sport participant in thevalid image, target image, or single image (hereinafter referred to asthe valid image or the like) selected in operation S7300.

As an example, when the valid image or the like selected in operationS7300 includes a single valid image, the location of the target sportparticipant may be determined in consideration of a pixel locationcorresponding to the target sport participant in the selected image.

As another example, when the valid image or the like selected inoperation S7300 includes a plurality of valid images, the location ofthe target sport participant may be determined in consideration of allor some pixel locations corresponding to the target sport participant inthe plurality of valid images or the like. For example, the location ofthe target sport participant may be determined by averaging the pixellocations acquired from the plurality of valid images or the like.

Alternatively, when the valid image or the like selected in operationS7300 includes a plurality of valid images, the location of the targetsport participant may be determined in further consideration of acondition related to the location of the target participant and alocation of a camera that captures the at least one valid image or acondition related to a lens distortion parameter. For example, thelocation of the target sport participant may be determined inconsideration of a pixel location corresponding to the target sportparticipant in a single image selected from the valid image or the likein consideration of a condition related to the location of the targetparticipant and the location of the camera that captures the at leastone valid image or a condition related to the lens distortion parameter.

As described above, although the description was focused on determiningthe location of the target sport participant on the basis of locationsacquired from image data, it is obvious that the details of FIGS. 16 to27 in which the location of the sport participant is determined inconsideration of locations acquired from positioning sensor data may beapplied.

The player tracking server 2000 according to an embodiment of thepresent disclosure will be described below.

According to an embodiment of the present disclosure, the playertracking server 2000 may acquire image data from the plurality of imagecapture devices 301, 302, 303, and 304 in order to determine a locationof a sport participant. Also, the player tracking server 2000 mayacquire positioning sensor data from the positioning sensor device 200in order to determine the location of the sport participant. Also, theplayer tracking server 2000 may compute an image-based location from theacquired image data and compute a positioning sensor-based location fromthe acquired positioning sensor data. Also, the player tracking server2000 may generate a virtual playfield or generate a matching table onthe basis of the computed image-based location and positioningsensor-based location. However, according to a preferred embodiment, theplayer tracking server 2000 may generate the virtual playfield orgenerate the matching table on the basis of the positioning sensor-basedlocation.

The details described with reference to FIG. 4 may be equally applied tothe player tracking server 2000 according to an embodiment of thepresent disclosure. That is, the player tracking server 2000 may includea communication module 1100, a memory 1200, and a controller 1300. Inparticular, the details described with reference to FIG. 4 may beequally applied to the communication module 1100, the memory 1200, andthe controller 1300, and thus a detailed description thereof will beomitted. Therefore, the following description will focus on detailsadded to the player tracking server 2000.

Some operations performed by an example of the player tracking system100 in association with the player tracking method will be described indetail below.

The player tracking system 100 according to this embodiment may acquirea plurality of pieces of image data from a plurality of image capturedevices. Specifically, the player tracking server 2000 may acquire aplurality of pieces of image data captured by a plurality of imagecapture devices. Also, the player tracking system 100 may acquirearrangement information related to the locations and orientations of theplurality of image capture devices from the plurality of image capturedevices. Also, the player tracking system 100 may acquire positioningsensor data from the positioning sensor device 200. Also, the playertracking system 100 may generate a virtual playfield and generate amatching table on the basis of positioning sensor data and arrangementinformation related to the image capture devices.

The player tracking server 2000 according to an embodiment of thepresent disclosure may determine the validity of the location of thesport participant acquired from the image data, which has been describedabove with reference to FIGS. 6 to 11 .

For example, the player tracking server 2000 may detect an occlusionevent in a plurality of pieces of image data acquired from the pluralityof image capture devices 301, 302, 303, and 304 or determine whether anocclusion event has occurred.

Also, the player tracking server 2000 may determine the severity of theocclusion event in the plurality of pieces of image data acquired fromthe plurality of image capture devices 301, 302, 303, and 304.

The player tracking system 100 according to this embodiment may generatea virtual playfield. Specifically, the player tracking server 2000 maybe provided to generate a virtual playfield on the basis of positioningsensor data and arrangement information related to the image capturedevices. Specifically, the player tracking server 2000 may generatevirtual points corresponding to the image capture devices on the basisof the arrangement information related to the locations of the imagecapture devices.

Also, the player tracking server 2000 may generate a plurality of movingpoints corresponding to a plurality of sport participants on the basisof positioning sensor data of the plurality of sport participants.

The player tracking system 100 according to this embodiment may beimplemented to predict an occlusion event (or the severity of anocclusion event) on the basis of the generated virtual playfield.Specifically, the player tracking server 2000 may be implemented topredict an occlusion event between sport participants on the basis ofarrangement information related to the orientations of the image capturedevices and a location relationship between the plurality of movingpoints in the virtual playfield.

The player tracking system 100 according to this embodiment may beimplemented to generate a matching table between the plurality of sportparticipants and the plurality of image capture devices on the basis ofthe generated virtual playfield.

Specifically, the player tracking server 2000 may compute a relationshipbetween the plurality of sport participants and the plurality of imagecapture devices on the basis of relative locations between a pluralityof moving points corresponding to the sport participants and a pluralityof points corresponding to the image capture devices in the generatedvirtual playfield. In this case, the player tracking server 2000 may beimplemented to generate a matching table between the sport participantsand the plurality of image capture devices on the basis of therelationship and a result of predicting the occlusion event.

The following description will refer to FIG. 29 . FIG. 29 is anexemplary diagram showing a virtual playfield according to an embodimentof the present disclosure.

The player tracking server 2000 may generate a virtual playfield on thebasis of location information of a plurality of sport participants andarrangement information of image capture devices.

The arrangement information of the image capture devices may includeinformation regarding locations, lines of sight, and orientations of theplurality of image capture devices 301, 302, 303, and 304.

The location information of the plurality of sport participants may beacquired from positioning sensor data. Alternatively, the locationinformation of the plurality of sport participants may be acquired fromimage data.

The player tracking server 2000 may generate a plurality of pointscorresponding to the plurality of image capture devices in a virtualplayfield coordinate system in consideration of the arrangementinformation of the image capture devices.

In this case, the locations of the image capture devices may be changedor fixed.

When the locations of the image capture devices are changed, the playertracking server 2000 may generate a plurality of moving pointscorresponding to the plurality of image capture devices in the virtualplayfield coordinate system in consideration of a change in the locationinformation of the plurality of image capture devices according to theprogress of a sport game.

When the locations of the image capture devices are fixed, the playertracking server 2000 may generate a plurality of fixed points FP1, FP2,FP3, and FP4 corresponding to the plurality of image capture devices inthe virtual playfield coordinate system as shown in FIG. 29 .

Also, the player tracking server 2000 may generate a plurality of pointsMP1, MP2, and MP3 corresponding to the plurality of sport participantsin a virtual playfield coordinate system in consideration of thelocation information of the plurality of sport participants.

Also, the player tracking server 2000 may generate a plurality of pointscorresponding to the plurality of sport participants in the virtualplayfield coordinate system in consideration of a change in the locationinformation of the plurality of sport participants according to theprogress of a sport game. That is, the plurality of points correspondingto the plurality of sport participants may be moving points MP1, MP2,and MP3.

Also, the player tracking server 2000 may compute a relationship betweenthe plurality of sport participants and the plurality of image capturedevices 301, 302, 303, and 304 on the basis of the generated virtualplayfield.

Specifically, the player tracking server 2000 may compute a relationshipbetween the plurality of sport participants and the plurality of imagecapture devices 301, 302, 303, and 304 on the basis of relativelocations between moving points MP1, MP2, and MP3 of the virtualplayfield corresponding to the plurality of sport participants andpoints FP1, FP2, FP3, and FP4 of the virtual playfield corresponding tothe plurality of image capture devices 301, 302, 303, and 304.

Also, the player tracking server 2000 may predict whether an occlusionevent has occurred between the plurality of sport participants on thebasis of the relationship between the plurality of sport participantsand the plurality of image capture devices 301, 302, 303, and 304.

For example, a plurality of virtual lines VL may be generated in thevirtual playfield from the points FP1, FP2, FP3, and FP4 correspondingto the plurality of image capture devices 301, 302, 303, and 304 to theplurality of moving points MP1, MP2, and MP3 corresponding to theplurality of sport participants.

In this case, the player tracking server 2000 may predict whether anocclusion event has occurred between sport participants in considerationof at least one of the locations of the points FP1, FP2, FP3, and FP4corresponding to the plurality of image capture devices 301, 302, 303,and 304, the locations of the plurality of moving points MP1, MP2, andMP3 corresponding to the plurality of sport participants, and anglesbetween the plurality of virtual lines VL1 and VL2.

For example, referring to FIG. 29 , the first virtual line VL1 extendingfrom the first fixed point FP1 to the first moving point MP1 and thesecond virtual line VL2 extending from the first fixed point FP1 to thesecond moving point MP2 may be generated.

In this case, when the first virtual line VL1 and the second virtualline VL2 overlap or when the angle between the first virtual line VL1and the second virtual line VL2 is less than a predetermined angle, theplayer tracking server 2000 may determine or predict that an occlusionevent is likely to have occurred between a first sport participantcorresponding to the first moving point MP1 and a second sportparticipant corresponding to the second moving point MP2.

In this case, additionally, the player tracking server 2000 maydetermine or predict that an occlusion event is likely to have occurredbetween the first sport participant corresponding to the first movingpoint MP1 and the second sport participant corresponding to the secondmoving point MP2 from the first image capture device 301 correspondingto the first fixed point FP1.

Also, the player tracking server 2000 may predict whether an occlusionevent occurs between sport participants in further consideration of adifference in location between the first moving point MP1 and the secondmoving point MP2.

For example, referring to FIG. 29 , even when the angle of the firstvirtual line VL1 is the same as the angle of the second virtual lineVL2, whether an occlusion has occurred between the first sportparticipant corresponding to the first moving point MP1 and the secondsport participant corresponding to the second moving point MP2 withrespect to the first fixed point FP1 may differ depending on thedifference in location between the first moving point MP1 and the secondmoving point MP2.

Therefore, the player tracking server 2000 may predict whether anocclusion event has occurred between sport participants in furtherconsideration of the difference in location between the moving pointscorresponding to the plurality of sport participants.

Also, the player tracking server 2000 may predict whether an occlusionevent has occurred between sport participants in consideration of thepoints of the virtual playfield corresponding to the plurality of imagecapture devices, the moving points of the virtual playfieldcorresponding to the plurality of sport participants, and informationregarding the lines of sight of the plurality of image capture devices.

For example, a virtual line corresponding to a line of sight may begenerated in consideration of a line of sight of the first image capturedevice 301 corresponding to the first fixed point FP1. In this case,when a first moving point and a second moving point are substantiallylocated on the virtual line corresponding to the line of sight, theplayer tracking server 2000 may determine that an occlusion event hasoccurred between a first sport participant corresponding to the firstmoving point and a second sport participant corresponding to the secondmoving point.

Also, the player tracking server 2000 may predict whether an occlusionevent has occurred between the plurality of sport participants later inconsideration of orientation information of the plurality of imagecapture devices and velocity data (or acceleration data) of theplurality of sport participants.

For example, referring to FIG. 29 , velocity data (or acceleration data)of the first sport participant and the third sport participant may beacquired by any suitable method. Also, orientation information of thefirst image capture device may be acquired. In this case, the playertracking server 2000 may predict that the first moving point MP1corresponding to the first sport participant moves toward the thirdmoving point MP3 corresponding to the third sport participant and thatthe third moving point MP3 corresponding to the third sport participantmoves toward the first moving point MP1 corresponding to the first sportparticipant on the basis of the velocity data (or the accelerationdata).

Also, the player tracking server 2000 may predict the orientation of thefirst fixed point FP1 corresponding to the first image capture device onthe basis of the orientation information.

Accordingly, the player tracking server 2000 may predict an occlusionbetween the first sport participant corresponding to the first movingpoint MP1 and the second port participant corresponding to the secondmoving point MP2 with respect to the first image capture devicecorresponding to the first fixed point FP1 later.

Also, the player tracking server 2000 may be implemented to generate amatching table between a plurality of sport participants and a pluralityof image capture devices on the basis of a relationship between theplurality of sport participants and the plurality of image capturedevices computed based on relative locations between a plurality ofmoving points corresponding to the plurality of sport participants and aplurality of points corresponding to the plurality of image capturedevices.

The player tracking server 2000 may generate a matching table betweenthe plurality of sport participants and the plurality of image capturedevices on the basis of the prediction of the occlusion.

As an example, virtual lines may extend from the plurality of fixedpoints FP1, FP2, FP3, and FP4 corresponding to the plurality of imagecapture devices 301, 302, 303, and 304 to the plurality of moving points(including MP1, MP2, and MP3) corresponding to the plurality of sportparticipants, and a table for a result of predicting the occlusion eventbased on angles between the virtual lines may be generated.

For example, referring to FIG. 29 , it may be predicted from the firstimage capture device 301 corresponding to the first fixed point FP1 thatan occlusion event has occurred between the first sport participantcorresponding to the first moving point MP1 and the second sportparticipant corresponding to the second moving point MP2.

On the other hand, it may be predicted from the second, third, andfourth image capture devices 302, 303, and 304 corresponding to thesecond, third, and fourth fixed points that no occlusion event hasoccurred between the first sport participant corresponding to the firstmoving point MP1 and the second sport participant corresponding to thesecond moving point MP2.

In this case, the player tracking server 2000 may generate a matchingtable on the basis of whether an occlusion is predicted based on thelocations of the plurality of fixed points FP1, FP2, FP3, and FP4corresponding to the image capture devices 301, 302, 303, and 304, thelocations of the moving points corresponding to the plurality of sportparticipants, and angles between the plurality of virtual lines.

Similarly, the player tracking server 2000 may generate a matching tableon the basis of whether an occlusion is predicted according to thedifference in location between the plurality of moving pointscorresponding to the plurality of sport participants.

Similarly, the player tracking server 2000 may generate a matching tableon the basis of whether an occlusion is predicted according to theorientation information of the plurality of image capture devices andthe velocity data (or acceleration data) of the plurality of sportparticipants.

The matching table generated as described above may be used to selectand continuously monitor an image capture device most suitable fortracking the target sport participant.

Also, the matching table may be used to select an image capture devicemost suitable for capturing the target sport participant in associationwith broadcasting.

Also, the matching table may be used to select an image capture devicemost suitable for generating an individual highlight of the target sportparticipant.

The configuration and operation of the player tracking server 2000according to an embodiment of the present disclosure have been describedabove. A player tracking method according to this embodiment will bedescribed below. In the following description, the player trackingmethod according to this embodiment is performed by the above-describedplayer tracking system 100. However, this is just for convenience ofdescription, and thus the player tracking method according to thisembodiment is not limited to the above-described player tracking system100. That is, the player tracking method, which will be described below,does not necessarily have to be performed only by the player trackingsystem 100 but may be performed by another system or device having afunction similar to that of the above-described player tracking system100.

The following description will refer to FIG. 30 . FIG. 30 is a flowchartillustrating a player tracking method using a plurality of image capturedevices according to an embodiment of the present disclosure.

Referring to FIG. 30 , a player tracking method using a plurality ofimage capture devices according to an embodiment of the presentdisclosure may include acquiring arrangement information of theplurality of image capture devices (S8100), receiving positioning sensordata of the plurality of sport participants (S8200), acquiring locationsof the plurality of sport participants from the positioning sensor data(S8300), generating a virtual playfield (S8400), and generating amatching table (S8500).

The above-described operations will be described in detail below.

The player tracking server 2000 may acquire the arrangement informationof the plurality of image capture devices (S8100). Specifically, inoperation S8100, the player tracking server 2000 may acquire a pluralityof pieces of image data captured by the plurality of image capturedevices 301, 302, 303, and 304 and arrangement information related tothe location and orientations of the plurality of image capture devices301, 302, 303, and 304 through the communication module 1100.

The player tracking server 2000 may receive the positioning sensor dataof the plurality of sport participants (S8200). Specifically, the playertracking server 2000 may acquire a plurality of pieces of image datafrom positioning sensor devices 200 worn by the plurality of sportparticipants through the communication module 1100.

In operation S8200, the player tracking server 2000 may acquirepositioning sensor data transmitted from the positioning sensor devices200 worn by the plurality of sport participants. The positioning sensordata may include sport participant-related location information, anidentifier of a sport participant, and the reliability of a sensorsignal.

The player tracking server 2000 may compute or acquire positioningsensor-based locations of the plurality of sport participants (S8300).Specifically, the player tracking server 2000 may acquire a plurality ofpieces of positioning sensor data from positioning sensor devices 200worn by the plurality of sport participants through the communicationmodule 1100. Specifically, the player tracking server 2000 may acquirelocations of the plurality of sport participants from the sportparticipant-related location information included in the positioningsensor data. Also, the player tracking server 2000 may acquire velocity-or acceleration-related information from the sport participant-relatedlocation information included in the positioning sensor data as well asthe acquired locations.

Also, the player tracking server 2000 may generate a virtual playfield(S8400). In this case, the virtual playfield may include a plurality ofpoints corresponding to the locations of the plurality of image capturedevices and a plurality of moving points corresponding to the locationsof the plurality of sport participants acquired in operation S8300.

The player tracking server 2000 may generate the plurality of pointscorresponding to the plurality of image capture devices on the basis ofarrangement information related to the locations of the plurality ofimage capture devices 301, 302, 303, and 304 acquired in operationS8100.

Also, the player tracking server 2000 may generate the plurality ofmoving points corresponding to the plurality of sport participants onthe basis of the locations of the plurality of sport participantsacquired in operation S8300.

Also, the player tracking server 2000 may generate a plurality ofvirtual lines on the basis of the plurality of moving pointscorresponding to the plurality of sport participants and the arrangementinformation related to the orientations of the plurality of imagecapture devices 301, 302, 303, and 304 acquired in operation S8100.

In operation S8400, the player tracking server 2000 may be provided tocompute a relationship between the plurality of sport participants andthe plurality of image capture devices.

For example, the player tracking server 2000 may compute a locationrelationship between the plurality of moving points corresponding to theplurality of sport participants and the plurality of fixed pointscorresponding to the image capture devices generated in the virtualplayfield.

As another example, the player tracking server 2000 may compute alocation relationship between the plurality of moving pointscorresponding to the plurality of sport participants generated in thevirtual playfield.

As another example, the player tracking server 2000 may compute arelationship such as an angle between the plurality of virtual linesgenerated in the virtual playfield.

Also, in operation S8400, the player tracking server 2000 may beimplemented to predict whether an occlusion event has occurred between aplurality of sport participants.

As an example, the player tracking server 2000 may predict whether anocclusion event has occurred between a plurality of sport participantson the basis of relative locations between the plurality of movingpoints corresponding to the plurality of sport participants and theplurality of virtual lines generated in the virtual playfield.

Also, the player tracking server 2000 may determine from which imagecapture device an occlusion event between the plurality of sportparticipants has occurred.

For example, referring to FIG. 29 again, the first virtual line VL1extending from the first fixed point FP1 to the first moving point MP1and the second virtual line VL2 extending from the first fixed point FP1to the second moving point MP2 may be generated.

In this case, when the first virtual line VL1 and the second virtualline VL2 overlap or when the angle between the first virtual line VL1and the second virtual line VL2 is less than a predetermined angle, theplayer tracking server 2000 may determine or predict that an occlusionevent is likely to have occurred between a first sport participantcorresponding to the first moving point MP1 and a second sportparticipant corresponding to the second moving point MP2.

In this case, additionally, the player tracking server 2000 maydetermine or predict that an occlusion event is likely to have occurredbetween the first sport participant corresponding to the first movingpoint MP1 and the second sport participant corresponding to the secondmoving point MP2 from the first image capture device 301 correspondingto the first fixed point FP1.

As another example, the player tracking server 2000 may predict whetheran occlusion event has occurred between a plurality of sportparticipants on the basis of a difference in location between theplurality of moving points corresponding to the plurality of sportparticipants.

For example, referring to FIG. 29 , even when the angle of the firstvirtual line VL1 is the same as the angle of the second virtual lineVL2, whether an occlusion has occurred between the first sportparticipant corresponding to the first moving point MP1 and the secondsport participant corresponding to the second moving point MP2 withrespect to the first fixed point FP1 may differ depending on thedifference in location between the first moving point MP1 and the secondmoving point MP2.

Therefore, the player tracking server 2000 may predict whether anocclusion event has occurred between sport participants in furtherconsideration of the difference in location between the moving pointscorresponding to the plurality of sport participants.

The player tracking server 2000 may generate a matching table (S8500).

The player tracking server 2000 may generate a matching table betweenthe plurality of sport participants and the plurality of image capturedevices on the basis of a result of predicting the occlusion between theplurality of sport participants, the relationship between the pluralityof sport participants, and the plurality of image capture devicescomputed in operation S8400.

As an example, virtual lines may extend from the plurality of fixedpoints FP1, FP2, FP3, and FP4 corresponding to the plurality of imagecapture devices 301, 302, 303, and 304 to the plurality of moving points(including MP1, MP2, and MP3) corresponding to the plurality of sportparticipants, and a table for a result of predicting the occlusion eventbased on angles between the virtual lines may be generated.

As another example, the player tracking server 2000 may generate amatching table on the basis of whether an occlusion is predictedaccording to the difference in location between the plurality of movingpoints corresponding to the plurality of sport participants.

As another example, the player tracking server 2000 may generate amatching table on the basis of whether an occlusion is predictedaccording to the orientation information of the plurality of imagecapture devices and the velocity data (or acceleration data) of theplurality of sport participants.

However, the above description is just an example, and the matchingtable may be generated by any suitable method.

The above-described player tracking method, player tracking device, andplayer tracking system using a plurality of image capture devicesaccording to an embodiment of the present disclosure can significantlyreduce the possibility of an error in computing the location of a sportparticipant due to an occlusion event compared to a case in which asingle image capture device is used.

The following description will refer to FIGS. 9 and 31 . FIG. 31 is aschematic diagram illustrating a method of computing a sportparticipant's location through a player tracking method using aplurality of image capture devices according to an embodiment of thepresent disclosure.

Referring to FIG. 9B, when a single image capture device is used, anerror due to a vertical movement of a sport participant 1 may occurbetween an actual location L2 and a location L2_image acquired fromimage data.

On the other hand, referring to FIG. 31 , when a plurality of imagecapture devices are used, the error due to the vertical movement of thesport participant 1 between the actual location L3 and the locationL3_image1 or L3_image2 acquired from the image data may be corrected.

For example, by considering the height of the plurality of image capturedevices and the angles of the plurality of image capture devices withrespect to the ground, a correction parameter for the location L3_image1or L3_image2 acquired from the image data may be computed. By applyingthe computed correction parameter to the location L3_image1 or L3_image2acquired from the image data, the actual location L3 of the sportparticipant 1 can be easily computed.

Also, in the case of using only a single image capture device, when anocclusion event occurs in image data acquired from the correspondingimage capture device, there is a lack of an alternative method foracquiring an accurate location using only image data.

On the other hand, in the case of using a plurality of image capturedevices, even when an occlusion event occurs in first image data, it ispossible to accurately compute the location of a sport participant usingsecond image data in which no occlusion event is detected.

Also, in the case of using the plurality of image capture devices, apanoramic view image or an image with improved quality may be generatedby matching a plurality of pieces of image data acquired from aplurality of image capture devices.

In this case, a panoramic view image or an image with improved qualitymay be generated by matching a plurality of pieces of image data on thebasis of references such as a reference point, a reference line, and areference surface of a region where the plurality of pieces of imagedata overlap.

The player tracking method, the player tracking device, and the playertracking system according to the present disclosure may be used toanalyze a sport participant. Specifically, the present disclosure may beapplied to all sport analysis fields, such as the analysis of themovement, velocity, and acceleration of a sport participant and tacticalanalysis, which require the location of a sport player to be accuratelycomputed.

Also, the above description has focused on soccer among sports, but thisis just an example. The present disclosure may be applied to all sportsthat require the location of a sport player to be computed.

Also, the player tracking method, the player tracking device, and theplayer tracking system according to the present disclosure are capableof continuously tracking a sport player, and thus may be used even invarious fields such as broadcasting related to video, video contentrelated to highlight generation, etc.

According to an embodiment of the present disclosure, by calculating aplayer's location by fusion of image data and positioning sensor data,it is possible to precisely track the player's location.

According to an embodiment of the present disclosure, by choosing imagedata optimized for a location of a player to be tracked using aplurality of cameras and calculating the location of the correspondingplayer from the image data, it is possible to precisely track theplayer's location.

The features, structures, and effects described in the above embodimentsare incorporated into at least one embodiment of the present disclosurebut are not necessarily limited to only one embodiment. Moreover,features, structures, and effects exemplified in one embodiment caneasily be combined and modified for another embodiment and then becarried out by those skilled in the art. Therefore, these combinationsand modifications should be construed as falling within the scope of thepresent disclosure.

While the present disclosure has been described with reference toembodiments, these are just examples and do not limit the presentdisclosure. It will be understood by those skilled in the art thatvarious modifications and applications may be made therein withoutdeparting from the essential characteristics of the embodiments. Thatis, elements described in the embodiments above in detail may bemodified. Furthermore, differences associated with such modificationsand applications should be construed as being included in the scope ofthe present disclosure defined by the appended claims.

What is claimed is:
 1. A multi-modal tracking method, the methodcomprising: receiving, from a plurality of positioning sensors eachbeing disposed at a plurality of sports participants, a plurality ofsensor signals each including a participant identifier and locationdata; receiving a sports image captured at a camera disposed peripheralto a playfield, the sports image including at least a target participantamong the plurality of sports participants in the playfield; detecting,from the sports image, an occlusion related to the target participant;determining a severity of the occlusion based on the sensor signalobtained from a specific positioning sensor disposed at specific sportsparticipants located in a region of interest related to the occlusion;when the occlusion of which the severity reflects a severe occlusion isnot detected, determining a location of the target participant using apixel position related to the target participant in the sports image;and when the occlusion of which the severity reflects a severe occlusionis detected, determining the location of the target participant usingthe location data of the sensor signal of which the participantidentifier corresponds to the target participant.
 2. The methodaccording to claim 1, wherein the region of interest is a predeterminedarea with reference to the target participant being involved in theocclusion, and wherein the determining the severity includes determiningthe severity of the occlusion based on a number of the at least onesensor signal located in the region of interest.
 3. The method accordingto claim 2, wherein the determining the severity includes: when thenumber of the at least one sensor signal is greater than a thresholdvalue, assigning a value reflecting the severe occlusion to theseverity, and when the number of the at least one sensor signal issmaller than the threshold value, assigning a value reflecting a mildocclusion to the severity.
 4. The method according to claim 1, whereinthe determining the severity includes: obtaining team informationindicating whether or not the target participant and anotherparticipant, among the at least one sport participant, who occludes oris occluded by the target participant are in a same team, based on theparticipant identifier corresponding to the target participant and theanother participant.
 5. The method according to claim 4, wherein thedetermining the severity includes: when the target participant and theanother participant are in the same team, assigning a value reflectingthe severe occlusion to the severity, when the target participant andthe another participant are in different teams, assigning a valuereflecting a mild occlusion to the severity.
 6. The method according toclaim 4, wherein the team information is obtained from pixel datacorresponding the target participant and the another participant of thesport image.